[==========] Running 14 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 12 tests from RaftConsensusElectionITest
[ RUN ] RaftConsensusElectionITest.TestNewLeaderCantResolvePeers
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260924 12:45:41.656059 5431 test_util.cc:268] Using random seed: -96466506
I20260924 12:45:41.667079 5431 ts_itest-base.cc:112] Starting cluster with:
I20260924 12:45:41.667156 5431 ts_itest-base.cc:113] --------------
I20260924 12:45:41.667196 5431 ts_itest-base.cc:114] 3 tablet servers
I20260924 12:45:41.667254 5431 ts_itest-base.cc:115] 3 replicas per TS
I20260924 12:45:41.667318 5431 ts_itest-base.cc:116] --------------
2026-09-24T12:45:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:45:41Z Disabled control of system clock
I20260924 12:45:41.688848 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.77.254:40069
--webserver_interface=127.5.77.254
--webserver_port=0
--builtin_ntp_servers=127.5.77.212:41239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.77.254:40069
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20260924 12:45:42.007665 5443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:45:42.007974 5443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:45:42.008091 5443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:45:42.015645 5443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:45:42.015761 5443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:45:42.015829 5443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:45:42.015899 5443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:45:42.024824 5443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:41239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.5.77.254:40069
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.77.254:40069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.77.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:45:42.026093 5443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:45:42.027907 5443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:45:42.039260 5451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:45:42.039213 5449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:45:42.039213 5448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:45:42.040009 5443 server_base.cc:1034] running on GCE node
I20260924 12:45:42.040772 5443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:45:42.042131 5443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:45:42.043370 5443 hybrid_clock.cc:648] HybridClock initialized: now 1790253942043243 us; error 88 us; skew 500 ppm
I20260924 12:45:42.043696 5443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:45:42.055342 5443 webserver.cc:460] Webserver started at http://127.5.77.254:40371/ using document root <none> and password file <none>
I20260924 12:45:42.056072 5443 fs_manager.cc:359] Metadata directory not provided
I20260924 12:45:42.056186 5443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:45:42.056484 5443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:45:42.059676 5443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/master-0/data/instance:
uuid: "ad93be810d2340f6bbdb06d8213717c9"
format_stamp: "Formatted at 2026-09-24 12:45:42 on dist-test-slave-b8q3"
I20260924 12:45:42.060406 5443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/master-0/wal/instance:
uuid: "ad93be810d2340f6bbdb06d8213717c9"
format_stamp: "Formatted at 2026-09-24 12:45:42 on dist-test-slave-b8q3"
I20260924 12:45:42.065505 5443 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 12:45:42.068919 5457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:42.070415 5443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 12:45:42.070637 5443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/master-0/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/master-0/wal
uuid: "ad93be810d2340f6bbdb06d8213717c9"
format_stamp: "Formatted at 2026-09-24 12:45:42 on dist-test-slave-b8q3"
I20260924 12:45:42.070828 5443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:45:42.092759 5443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:45:42.094012 5443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:45:42.094303 5443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:45:42.112140 5443 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.254:40069
I20260924 12:45:42.112169 5508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.254:40069 every 8 connection(s)
I20260924 12:45:42.113481 5443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/master-0/data/info.pb
I20260924 12:45:42.117169 5509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:45:42.122279 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 5443
I20260924 12:45:42.122557 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/master-0/wal/instance
I20260924 12:45:42.129009 5509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad93be810d2340f6bbdb06d8213717c9: Bootstrap starting.
I20260924 12:45:42.132194 5509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad93be810d2340f6bbdb06d8213717c9: Neither blocks nor log segments found. Creating new log.
I20260924 12:45:42.133126 5509 log.cc:824] T 00000000000000000000000000000000 P ad93be810d2340f6bbdb06d8213717c9: Log is configured to *not* fsync() on all Append() calls
I20260924 12:45:42.135605 5509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad93be810d2340f6bbdb06d8213717c9: No bootstrap required, opened a new log
I20260924 12:45:42.145789 5509 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad93be810d2340f6bbdb06d8213717c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad93be810d2340f6bbdb06d8213717c9" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 40069 } }
I20260924 12:45:42.146155 5509 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad93be810d2340f6bbdb06d8213717c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:45:42.146248 5509 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad93be810d2340f6bbdb06d8213717c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad93be810d2340f6bbdb06d8213717c9, State: Initialized, Role: FOLLOWER
I20260924 12:45:42.146840 5509 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad93be810d2340f6bbdb06d8213717c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad93be810d2340f6bbdb06d8213717c9" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 40069 } }
I20260924 12:45:42.147101 5509 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad93be810d2340f6bbdb06d8213717c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:45:42.147229 5509 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad93be810d2340f6bbdb06d8213717c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:45:42.147435 5509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad93be810d2340f6bbdb06d8213717c9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:45:42.149757 5509 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad93be810d2340f6bbdb06d8213717c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad93be810d2340f6bbdb06d8213717c9" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 40069 } }
I20260924 12:45:42.150247 5509 leader_election.cc:302] T 00000000000000000000000000000000 P ad93be810d2340f6bbdb06d8213717c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad93be810d2340f6bbdb06d8213717c9; no voters:
I20260924 12:45:42.150938 5509 leader_election.cc:288] T 00000000000000000000000000000000 P ad93be810d2340f6bbdb06d8213717c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:45:42.151084 5514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad93be810d2340f6bbdb06d8213717c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:45:42.154063 5509 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad93be810d2340f6bbdb06d8213717c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:45:42.154270 5514 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad93be810d2340f6bbdb06d8213717c9 [term 1 LEADER]: Becoming Leader. State: Replica: ad93be810d2340f6bbdb06d8213717c9, State: Running, Role: LEADER
I20260924 12:45:42.154896 5514 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad93be810d2340f6bbdb06d8213717c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad93be810d2340f6bbdb06d8213717c9" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 40069 } }
I20260924 12:45:42.157861 5516 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad93be810d2340f6bbdb06d8213717c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad93be810d2340f6bbdb06d8213717c9. Latest consensus state: current_term: 1 leader_uuid: "ad93be810d2340f6bbdb06d8213717c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad93be810d2340f6bbdb06d8213717c9" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 40069 } } }
I20260924 12:45:42.158195 5516 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad93be810d2340f6bbdb06d8213717c9 [sys.catalog]: This master's current role is: LEADER
I20260924 12:45:42.161599 5515 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad93be810d2340f6bbdb06d8213717c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad93be810d2340f6bbdb06d8213717c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad93be810d2340f6bbdb06d8213717c9" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 40069 } } }
I20260924 12:45:42.161840 5515 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad93be810d2340f6bbdb06d8213717c9 [sys.catalog]: This master's current role is: LEADER
I20260924 12:45:42.164139 5523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:45:42.168202 5523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:45:42.176368 5523 catalog_manager.cc:1347] Generated new cluster ID: ebbe86326c2f456db966b71d457bf1ec
I20260924 12:45:42.176478 5523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:45:42.185316 5523 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:45:42.187152 5523 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:45:42.206553 5523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad93be810d2340f6bbdb06d8213717c9: Generated new TSK 0
I20260924 12:45:42.207465 5523 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:45:42.215152 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.193:0
--local_ip_for_outbound_sockets=127.5.77.193
--webserver_interface=127.5.77.193
--webserver_port=0
--tserver_master_addrs=127.5.77.254:40069
--builtin_ntp_servers=127.5.77.212:41239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--dns_resolver_cache_capacity_mb=0
--enable_leader_failure_detection=false
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=3
--leader_failure_max_missed_heartbeat_periods=10
--leader_failure_exp_backoff_max_delta_ms=5000 with env {}
W20260924 12:45:42.543027 5533 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 12:45:42.543344 5533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:45:42.543430 5533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:45:42.543522 5533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:45:42.550741 5533 flags.cc:425] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=5000
W20260924 12:45:42.550849 5533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:45:42.550995 5533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.193
I20260924 12:45:42.560334 5533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:41239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--enable_leader_failure_detection=false
--leader_failure_exp_backoff_max_delta_ms=5000
--leader_failure_max_missed_heartbeat_periods=10
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.77.193
--webserver_port=0
--tserver_master_addrs=127.5.77.254:40069
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--dns_resolver_cache_capacity_mb=0
--local_ip_for_outbound_sockets=127.5.77.193
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:45:42.561615 5533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:45:42.563436 5533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:45:42.576020 5538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:45:42.576066 5539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:45:42.576134 5541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:45:42.576874 5533 server_base.cc:1034] running on GCE node
I20260924 12:45:42.577445 5533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:45:42.578383 5533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:45:42.579658 5533 hybrid_clock.cc:648] HybridClock initialized: now 1790253942579565 us; error 42 us; skew 500 ppm
I20260924 12:45:42.580034 5533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:45:42.582257 5533 webserver.cc:460] Webserver started at http://127.5.77.193:36297/ using document root <none> and password file <none>
I20260924 12:45:42.582948 5533 fs_manager.cc:359] Metadata directory not provided
I20260924 12:45:42.583045 5533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:45:42.583281 5533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:45:42.585757 5533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-0/data/instance:
uuid: "e62a7464cd9147448d8e7a8087cfb6ff"
format_stamp: "Formatted at 2026-09-24 12:45:42 on dist-test-slave-b8q3"
I20260924 12:45:42.586437 5533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-0/wal/instance:
uuid: "e62a7464cd9147448d8e7a8087cfb6ff"
format_stamp: "Formatted at 2026-09-24 12:45:42 on dist-test-slave-b8q3"
I20260924 12:45:42.591742 5533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 12:45:42.595045 5547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:42.596432 5533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:45:42.596576 5533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-0/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-0/wal
uuid: "e62a7464cd9147448d8e7a8087cfb6ff"
format_stamp: "Formatted at 2026-09-24 12:45:42 on dist-test-slave-b8q3"
I20260924 12:45:42.596757 5533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:45:42.623219 5533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:45:42.624042 5533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:45:42.624286 5533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:45:42.625156 5533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:45:42.626953 5533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:45:42.627027 5533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:42.627116 5533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:45:42.627157 5533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:42.657701 5533 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.193:41341
I20260924 12:45:42.657745 5659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.193:41341 every 8 connection(s)
I20260924 12:45:42.659013 5533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-0/data/info.pb
I20260924 12:45:42.660449 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 5533
I20260924 12:45:42.660621 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-0/wal/instance
I20260924 12:45:42.663925 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.194:0
--local_ip_for_outbound_sockets=127.5.77.194
--webserver_interface=127.5.77.194
--webserver_port=0
--tserver_master_addrs=127.5.77.254:40069
--builtin_ntp_servers=127.5.77.212:41239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--dns_resolver_cache_capacity_mb=0
--enable_leader_failure_detection=false
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=3
--leader_failure_max_missed_heartbeat_periods=10
--leader_failure_exp_backoff_max_delta_ms=5000 with env {}
I20260924 12:45:42.681003 5660 heartbeater.cc:347] Connected to a master server at 127.5.77.254:40069
I20260924 12:45:42.681291 5660 heartbeater.cc:464] Registering TS with master...
I20260924 12:45:42.681978 5660 heartbeater.cc:511] Master 127.5.77.254:40069 requested a full tablet report, sending...
I20260924 12:45:42.683998 5474 ts_manager.cc:194] Registered new tserver with Master: e62a7464cd9147448d8e7a8087cfb6ff (127.5.77.193:41341)
I20260924 12:45:42.685508 5474 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.193:43809
W20260924 12:45:42.989426 5664 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 12:45:42.989754 5664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:45:42.989841 5664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:45:42.989941 5664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:45:42.997817 5664 flags.cc:425] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=5000
W20260924 12:45:42.997936 5664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:45:42.998093 5664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.194
I20260924 12:45:43.007570 5664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:41239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--enable_leader_failure_detection=false
--leader_failure_exp_backoff_max_delta_ms=5000
--leader_failure_max_missed_heartbeat_periods=10
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.77.194
--webserver_port=0
--tserver_master_addrs=127.5.77.254:40069
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--dns_resolver_cache_capacity_mb=0
--local_ip_for_outbound_sockets=127.5.77.194
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:45:43.008836 5664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:45:43.010562 5664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:45:43.022831 5669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:45:43.022776 5670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:45:43.022886 5672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:45:43.023442 5664 server_base.cc:1034] running on GCE node
I20260924 12:45:43.023996 5664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:45:43.024855 5664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:45:43.026069 5664 hybrid_clock.cc:648] HybridClock initialized: now 1790253943026001 us; error 39 us; skew 500 ppm
I20260924 12:45:43.026361 5664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:45:43.028496 5664 webserver.cc:460] Webserver started at http://127.5.77.194:33711/ using document root <none> and password file <none>
I20260924 12:45:43.029135 5664 fs_manager.cc:359] Metadata directory not provided
I20260924 12:45:43.029253 5664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:45:43.029510 5664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:45:43.032449 5664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-1/data/instance:
uuid: "2ffa41fc303343f587ab7d5ef638a311"
format_stamp: "Formatted at 2026-09-24 12:45:43 on dist-test-slave-b8q3"
I20260924 12:45:43.033145 5664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-1/wal/instance:
uuid: "2ffa41fc303343f587ab7d5ef638a311"
format_stamp: "Formatted at 2026-09-24 12:45:43 on dist-test-slave-b8q3"
I20260924 12:45:43.038005 5664 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 12:45:43.041488 5678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:43.042904 5664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:45:43.043054 5664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-1/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-1/wal
uuid: "2ffa41fc303343f587ab7d5ef638a311"
format_stamp: "Formatted at 2026-09-24 12:45:43 on dist-test-slave-b8q3"
I20260924 12:45:43.043243 5664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:45:43.060266 5664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:45:43.061084 5664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:45:43.061344 5664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:45:43.062671 5664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:45:43.064102 5664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:45:43.064194 5664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:43.064297 5664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:45:43.064350 5664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:43.095584 5664 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.194:37943
I20260924 12:45:43.095618 5790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.194:37943 every 8 connection(s)
I20260924 12:45:43.096808 5664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-1/data/info.pb
I20260924 12:45:43.100371 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 5664
I20260924 12:45:43.100528 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-1/wal/instance
I20260924 12:45:43.104044 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.195:0
--local_ip_for_outbound_sockets=127.5.77.195
--webserver_interface=127.5.77.195
--webserver_port=0
--tserver_master_addrs=127.5.77.254:40069
--builtin_ntp_servers=127.5.77.212:41239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--dns_resolver_cache_capacity_mb=0
--enable_leader_failure_detection=false
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=3
--leader_failure_max_missed_heartbeat_periods=10
--leader_failure_exp_backoff_max_delta_ms=5000 with env {}
I20260924 12:45:43.111964 5791 heartbeater.cc:347] Connected to a master server at 127.5.77.254:40069
I20260924 12:45:43.112291 5791 heartbeater.cc:464] Registering TS with master...
I20260924 12:45:43.113158 5791 heartbeater.cc:511] Master 127.5.77.254:40069 requested a full tablet report, sending...
I20260924 12:45:43.114804 5474 ts_manager.cc:194] Registered new tserver with Master: 2ffa41fc303343f587ab7d5ef638a311 (127.5.77.194:37943)
I20260924 12:45:43.115904 5474 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.194:43987
W20260924 12:45:43.432097 5794 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 12:45:43.432428 5794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:45:43.432516 5794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:45:43.432618 5794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:45:43.440507 5794 flags.cc:425] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=5000
W20260924 12:45:43.440630 5794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:45:43.440789 5794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.195
I20260924 12:45:43.450129 5794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:41239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--enable_leader_failure_detection=false
--leader_failure_exp_backoff_max_delta_ms=5000
--leader_failure_max_missed_heartbeat_periods=10
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.77.195
--webserver_port=0
--tserver_master_addrs=127.5.77.254:40069
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--dns_resolver_cache_capacity_mb=0
--local_ip_for_outbound_sockets=127.5.77.195
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:45:43.451462 5794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:45:43.453229 5794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:45:43.465970 5799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:45:43.465996 5802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:45:43.466055 5800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:45:43.467054 5794 server_base.cc:1034] running on GCE node
I20260924 12:45:43.467614 5794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:45:43.468530 5794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:45:43.469738 5794 hybrid_clock.cc:648] HybridClock initialized: now 1790253943469667 us; error 50 us; skew 500 ppm
I20260924 12:45:43.470010 5794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:45:43.472342 5794 webserver.cc:460] Webserver started at http://127.5.77.195:42009/ using document root <none> and password file <none>
I20260924 12:45:43.472987 5794 fs_manager.cc:359] Metadata directory not provided
I20260924 12:45:43.473083 5794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:45:43.473320 5794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:45:43.475999 5794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-2/data/instance:
uuid: "7eee479d28d44f5f842fedc75fde9550"
format_stamp: "Formatted at 2026-09-24 12:45:43 on dist-test-slave-b8q3"
I20260924 12:45:43.476691 5794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-2/wal/instance:
uuid: "7eee479d28d44f5f842fedc75fde9550"
format_stamp: "Formatted at 2026-09-24 12:45:43 on dist-test-slave-b8q3"
I20260924 12:45:43.482242 5794 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 12:45:43.485785 5808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:43.487305 5794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:45:43.487442 5794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-2/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-2/wal
uuid: "7eee479d28d44f5f842fedc75fde9550"
format_stamp: "Formatted at 2026-09-24 12:45:43 on dist-test-slave-b8q3"
I20260924 12:45:43.487614 5794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:45:43.505005 5794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:45:43.505829 5794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:45:43.506072 5794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:45:43.506991 5794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:45:43.508399 5794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:45:43.508471 5794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:43.508560 5794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:45:43.508601 5794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:43.542047 5794 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.195:33047
I20260924 12:45:43.542076 5920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.195:33047 every 8 connection(s)
I20260924 12:45:43.543339 5794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-2/data/info.pb
I20260924 12:45:43.551255 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 5794
I20260924 12:45:43.551429 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-2/wal/instance
I20260924 12:45:43.556479 5921 heartbeater.cc:347] Connected to a master server at 127.5.77.254:40069
I20260924 12:45:43.556798 5921 heartbeater.cc:464] Registering TS with master...
I20260924 12:45:43.557462 5921 heartbeater.cc:511] Master 127.5.77.254:40069 requested a full tablet report, sending...
I20260924 12:45:43.558812 5474 ts_manager.cc:194] Registered new tserver with Master: 7eee479d28d44f5f842fedc75fde9550 (127.5.77.195:33047)
I20260924 12:45:43.559602 5474 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.195:41353
I20260924 12:45:43.567878 5431 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:45:43.571743 5431 test_util.cc:268] Using random seed: -94550797
I20260924 12:45:43.597148 5474 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44254:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:45:43.598793 5474 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:45:43.626400 5856 tablet_service.cc:1467] Processing CreateTablet for tablet 2e6e1d30fc99418ebeca18a7394a2662 (DEFAULT_TABLE table=test-workload [id=8f068027a9be4f6e9e75b857afc47d0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:45:43.626729 5595 tablet_service.cc:1467] Processing CreateTablet for tablet 2e6e1d30fc99418ebeca18a7394a2662 (DEFAULT_TABLE table=test-workload [id=8f068027a9be4f6e9e75b857afc47d0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:45:43.627843 5595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e6e1d30fc99418ebeca18a7394a2662. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:45:43.627995 5856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e6e1d30fc99418ebeca18a7394a2662. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:45:43.627905 5726 tablet_service.cc:1467] Processing CreateTablet for tablet 2e6e1d30fc99418ebeca18a7394a2662 (DEFAULT_TABLE table=test-workload [id=8f068027a9be4f6e9e75b857afc47d0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:45:43.629226 5726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e6e1d30fc99418ebeca18a7394a2662. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:45:43.638089 5939 tablet_bootstrap.cc:492] T 2e6e1d30fc99418ebeca18a7394a2662 P e62a7464cd9147448d8e7a8087cfb6ff: Bootstrap starting.
I20260924 12:45:43.641538 5939 tablet_bootstrap.cc:654] T 2e6e1d30fc99418ebeca18a7394a2662 P e62a7464cd9147448d8e7a8087cfb6ff: Neither blocks nor log segments found. Creating new log.
I20260924 12:45:43.643510 5940 tablet_bootstrap.cc:492] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550: Bootstrap starting.
I20260924 12:45:43.643961 5939 log.cc:824] T 2e6e1d30fc99418ebeca18a7394a2662 P e62a7464cd9147448d8e7a8087cfb6ff: Log is configured to *not* fsync() on all Append() calls
I20260924 12:45:43.644466 5941 tablet_bootstrap.cc:492] T 2e6e1d30fc99418ebeca18a7394a2662 P 2ffa41fc303343f587ab7d5ef638a311: Bootstrap starting.
I20260924 12:45:43.647413 5941 tablet_bootstrap.cc:654] T 2e6e1d30fc99418ebeca18a7394a2662 P 2ffa41fc303343f587ab7d5ef638a311: Neither blocks nor log segments found. Creating new log.
I20260924 12:45:43.647569 5940 tablet_bootstrap.cc:654] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550: Neither blocks nor log segments found. Creating new log.
I20260924 12:45:43.648473 5941 log.cc:824] T 2e6e1d30fc99418ebeca18a7394a2662 P 2ffa41fc303343f587ab7d5ef638a311: Log is configured to *not* fsync() on all Append() calls
I20260924 12:45:43.648862 5940 log.cc:824] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550: Log is configured to *not* fsync() on all Append() calls
I20260924 12:45:43.651319 5941 tablet_bootstrap.cc:492] T 2e6e1d30fc99418ebeca18a7394a2662 P 2ffa41fc303343f587ab7d5ef638a311: No bootstrap required, opened a new log
I20260924 12:45:43.651651 5941 ts_tablet_manager.cc:1397] T 2e6e1d30fc99418ebeca18a7394a2662 P 2ffa41fc303343f587ab7d5ef638a311: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260924 12:45:43.651865 5939 tablet_bootstrap.cc:492] T 2e6e1d30fc99418ebeca18a7394a2662 P e62a7464cd9147448d8e7a8087cfb6ff: No bootstrap required, opened a new log
I20260924 12:45:43.652122 5939 ts_tablet_manager.cc:1397] T 2e6e1d30fc99418ebeca18a7394a2662 P e62a7464cd9147448d8e7a8087cfb6ff: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.002s
I20260924 12:45:43.652693 5940 tablet_bootstrap.cc:492] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550: No bootstrap required, opened a new log
I20260924 12:45:43.652949 5940 ts_tablet_manager.cc:1397] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260924 12:45:43.660663 5941 raft_consensus.cc:348] T 2e6e1d30fc99418ebeca18a7394a2662 P 2ffa41fc303343f587ab7d5ef638a311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eee479d28d44f5f842fedc75fde9550" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33047 } } peers { permanent_uuid: "e62a7464cd9147448d8e7a8087cfb6ff" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41341 } } peers { permanent_uuid: "2ffa41fc303343f587ab7d5ef638a311" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 37943 } }
I20260924 12:45:43.661068 5941 raft_consensus.cc:729] T 2e6e1d30fc99418ebeca18a7394a2662 P 2ffa41fc303343f587ab7d5ef638a311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ffa41fc303343f587ab7d5ef638a311, State: Initialized, Role: FOLLOWER
I20260924 12:45:43.661499 5941 consensus_queue.cc:260] T 2e6e1d30fc99418ebeca18a7394a2662 P 2ffa41fc303343f587ab7d5ef638a311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eee479d28d44f5f842fedc75fde9550" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33047 } } peers { permanent_uuid: "e62a7464cd9147448d8e7a8087cfb6ff" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41341 } } peers { permanent_uuid: "2ffa41fc303343f587ab7d5ef638a311" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 37943 } }
I20260924 12:45:43.663312 5941 ts_tablet_manager.cc:1428] T 2e6e1d30fc99418ebeca18a7394a2662 P 2ffa41fc303343f587ab7d5ef638a311: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260924 12:45:43.663897 5940 raft_consensus.cc:348] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eee479d28d44f5f842fedc75fde9550" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33047 } } peers { permanent_uuid: "e62a7464cd9147448d8e7a8087cfb6ff" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41341 } } peers { permanent_uuid: "2ffa41fc303343f587ab7d5ef638a311" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 37943 } }
I20260924 12:45:43.664336 5940 raft_consensus.cc:729] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eee479d28d44f5f842fedc75fde9550, State: Initialized, Role: FOLLOWER
I20260924 12:45:43.664606 5791 heartbeater.cc:503] Master 127.5.77.254:40069 was elected leader, sending a full tablet report...
I20260924 12:45:43.664860 5940 consensus_queue.cc:260] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eee479d28d44f5f842fedc75fde9550" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33047 } } peers { permanent_uuid: "e62a7464cd9147448d8e7a8087cfb6ff" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41341 } } peers { permanent_uuid: "2ffa41fc303343f587ab7d5ef638a311" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 37943 } }
I20260924 12:45:43.665918 5921 heartbeater.cc:503] Master 127.5.77.254:40069 was elected leader, sending a full tablet report...
I20260924 12:45:43.666575 5940 ts_tablet_manager.cc:1428] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260924 12:45:43.669718 5939 raft_consensus.cc:348] T 2e6e1d30fc99418ebeca18a7394a2662 P e62a7464cd9147448d8e7a8087cfb6ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eee479d28d44f5f842fedc75fde9550" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33047 } } peers { permanent_uuid: "e62a7464cd9147448d8e7a8087cfb6ff" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41341 } } peers { permanent_uuid: "2ffa41fc303343f587ab7d5ef638a311" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 37943 } }
I20260924 12:45:43.670176 5939 raft_consensus.cc:729] T 2e6e1d30fc99418ebeca18a7394a2662 P e62a7464cd9147448d8e7a8087cfb6ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e62a7464cd9147448d8e7a8087cfb6ff, State: Initialized, Role: FOLLOWER
I20260924 12:45:43.670636 5939 consensus_queue.cc:260] T 2e6e1d30fc99418ebeca18a7394a2662 P e62a7464cd9147448d8e7a8087cfb6ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eee479d28d44f5f842fedc75fde9550" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33047 } } peers { permanent_uuid: "e62a7464cd9147448d8e7a8087cfb6ff" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41341 } } peers { permanent_uuid: "2ffa41fc303343f587ab7d5ef638a311" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 37943 } }
I20260924 12:45:43.675144 5660 heartbeater.cc:503] Master 127.5.77.254:40069 was elected leader, sending a full tablet report...
I20260924 12:45:43.677166 5939 ts_tablet_manager.cc:1428] T 2e6e1d30fc99418ebeca18a7394a2662 P e62a7464cd9147448d8e7a8087cfb6ff: Time spent starting tablet: real 0.025s user 0.008s sys 0.008s
W20260924 12:45:43.711905 5786 generic_service.cc:139] {username='slave'} at 127.0.0.1:33866 forcing change of non-runtime-safe flag fail_dns_resolution_hostports
I20260924 12:45:43.712141 5786 generic_service.cc:175] {username='slave'} at 127.0.0.1:33866 changed flags via RPC: fail_dns_resolution_hostports from '' to '127.5.77.194:37943'
W20260924 12:45:43.713644 5786 generic_service.cc:139] {username='slave'} at 127.0.0.1:33866 forcing change of non-runtime-safe flag fail_dns_resolution
I20260924 12:45:43.713743 5786 generic_service.cc:175] {username='slave'} at 127.0.0.1:33866 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
W20260924 12:45:43.720523 5916 generic_service.cc:139] {username='slave'} at 127.0.0.1:51922 forcing change of non-runtime-safe flag fail_dns_resolution_hostports
I20260924 12:45:43.720798 5916 generic_service.cc:175] {username='slave'} at 127.0.0.1:51922 changed flags via RPC: fail_dns_resolution_hostports from '' to '127.5.77.194:37943'
W20260924 12:45:43.722101 5916 generic_service.cc:139] {username='slave'} at 127.0.0.1:51922 forcing change of non-runtime-safe flag fail_dns_resolution
I20260924 12:45:43.722203 5916 generic_service.cc:175] {username='slave'} at 127.0.0.1:51922 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
W20260924 12:45:43.729054 5655 generic_service.cc:139] {username='slave'} at 127.0.0.1:36542 forcing change of non-runtime-safe flag fail_dns_resolution_hostports
I20260924 12:45:43.729315 5655 generic_service.cc:175] {username='slave'} at 127.0.0.1:36542 changed flags via RPC: fail_dns_resolution_hostports from '' to '127.5.77.194:37943'
W20260924 12:45:43.730955 5655 generic_service.cc:139] {username='slave'} at 127.0.0.1:36542 forcing change of non-runtime-safe flag fail_dns_resolution
I20260924 12:45:43.731060 5655 generic_service.cc:175] {username='slave'} at 127.0.0.1:36542 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
I20260924 12:45:43.732436 5876 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2e6e1d30fc99418ebeca18a7394a2662"
dest_uuid: "7eee479d28d44f5f842fedc75fde9550"
from {username='slave'} at 127.0.0.1:51910
I20260924 12:45:43.732743 5876 raft_consensus.cc:482] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 12:45:43.732909 5876 raft_consensus.cc:3037] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 [term 0 FOLLOWER]: Advancing to term 1
W20260924 12:45:43.794884 5922 tablet.cc:2369] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:45:43.818159 5876 raft_consensus.cc:504] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eee479d28d44f5f842fedc75fde9550" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33047 } } peers { permanent_uuid: "e62a7464cd9147448d8e7a8087cfb6ff" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41341 } } peers { permanent_uuid: "2ffa41fc303343f587ab7d5ef638a311" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 37943 } }
I20260924 12:45:43.820384 5876 leader_election.cc:288] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 [CANDIDATE]: Term 1 election: Requested vote from peers e62a7464cd9147448d8e7a8087cfb6ff (127.5.77.193:41341), 2ffa41fc303343f587ab7d5ef638a311 (127.5.77.194:37943)
W20260924 12:45:43.821586 5950 leader_election.cc:334] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 2ffa41fc303343f587ab7d5ef638a311 (127.5.77.194:37943): Network error: failed to refresh physical address: injected DNS resolution failure: 127.5.77.194:37943
I20260924 12:45:43.827005 5615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e6e1d30fc99418ebeca18a7394a2662" candidate_uuid: "7eee479d28d44f5f842fedc75fde9550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e62a7464cd9147448d8e7a8087cfb6ff"
I20260924 12:45:43.827400 5615 raft_consensus.cc:3037] T 2e6e1d30fc99418ebeca18a7394a2662 P e62a7464cd9147448d8e7a8087cfb6ff [term 0 FOLLOWER]: Advancing to term 1
W20260924 12:45:43.848791 5792 tablet.cc:2369] T 2e6e1d30fc99418ebeca18a7394a2662 P 2ffa41fc303343f587ab7d5ef638a311: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 12:45:43.911507 5661 tablet.cc:2369] T 2e6e1d30fc99418ebeca18a7394a2662 P e62a7464cd9147448d8e7a8087cfb6ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:45:43.919052 5615 raft_consensus.cc:2454] T 2e6e1d30fc99418ebeca18a7394a2662 P e62a7464cd9147448d8e7a8087cfb6ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eee479d28d44f5f842fedc75fde9550 in term 1.
I20260924 12:45:43.919900 5809 leader_election.cc:302] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7eee479d28d44f5f842fedc75fde9550, e62a7464cd9147448d8e7a8087cfb6ff; no voters: 2ffa41fc303343f587ab7d5ef638a311
I20260924 12:45:43.920395 5946 raft_consensus.cc:2781] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:45:43.920991 5946 raft_consensus.cc:686] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 [term 1 LEADER]: Becoming Leader. State: Replica: 7eee479d28d44f5f842fedc75fde9550, State: Running, Role: LEADER
I20260924 12:45:43.921617 5946 consensus_queue.cc:237] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eee479d28d44f5f842fedc75fde9550" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33047 } } peers { permanent_uuid: "e62a7464cd9147448d8e7a8087cfb6ff" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41341 } } peers { permanent_uuid: "2ffa41fc303343f587ab7d5ef638a311" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 37943 } }
I20260924 12:45:43.926676 5474 catalog_manager.cc:5495] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eee479d28d44f5f842fedc75fde9550 (127.5.77.195). New cstate: current_term: 1 leader_uuid: "7eee479d28d44f5f842fedc75fde9550" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eee479d28d44f5f842fedc75fde9550" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e62a7464cd9147448d8e7a8087cfb6ff" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ffa41fc303343f587ab7d5ef638a311" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 37943 } health_report { overall_health: UNKNOWN } } }
W20260924 12:45:44.027654 5950 consensus_peers.cc:487] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 -> Peer 2ffa41fc303343f587ab7d5ef638a311 (127.5.77.194:37943): Couldn't send request to peer 2ffa41fc303343f587ab7d5ef638a311. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.5.77.194:37943. This is attempt 1: this message will repeat every 5th retry.
I20260924 12:45:44.043309 5946 consensus_queue.cc:1035] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e62a7464cd9147448d8e7a8087cfb6ff" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 12:45:44.505138 5950 consensus_peers.cc:487] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 -> Peer 2ffa41fc303343f587ab7d5ef638a311 (127.5.77.194:37943): Couldn't send request to peer 2ffa41fc303343f587ab7d5ef638a311. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.5.77.194:37943. This is attempt 6: this message will repeat every 5th retry.
W20260924 12:45:44.987368 5950 consensus_peers.cc:487] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 -> Peer 2ffa41fc303343f587ab7d5ef638a311 (127.5.77.194:37943): Couldn't send request to peer 2ffa41fc303343f587ab7d5ef638a311. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.5.77.194:37943. This is attempt 11: this message will repeat every 5th retry.
W20260924 12:45:45.508705 5950 consensus_peers.cc:487] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 -> Peer 2ffa41fc303343f587ab7d5ef638a311 (127.5.77.194:37943): Couldn't send request to peer 2ffa41fc303343f587ab7d5ef638a311. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.5.77.194:37943. This is attempt 16: this message will repeat every 5th retry.
W20260924 12:45:46.005810 5950 consensus_peers.cc:487] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 -> Peer 2ffa41fc303343f587ab7d5ef638a311 (127.5.77.194:37943): Couldn't send request to peer 2ffa41fc303343f587ab7d5ef638a311. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.5.77.194:37943. This is attempt 21: this message will repeat every 5th retry.
W20260924 12:45:46.468146 5950 consensus_peers.cc:487] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 -> Peer 2ffa41fc303343f587ab7d5ef638a311 (127.5.77.194:37943): Couldn't send request to peer 2ffa41fc303343f587ab7d5ef638a311. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.5.77.194:37943. This is attempt 26: this message will repeat every 5th retry.
I20260924 12:45:47.004632 5946 consensus_queue.cc:579] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 [LEADER]: Leader has been unable to successfully communicate with peer 2ffa41fc303343f587ab7d5ef638a311 for more than 3 seconds (3.081s)
W20260924 12:45:47.005520 5950 consensus_peers.cc:487] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 -> Peer 2ffa41fc303343f587ab7d5ef638a311 (127.5.77.194:37943): Couldn't send request to peer 2ffa41fc303343f587ab7d5ef638a311. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.5.77.194:37943. This is attempt 31: this message will repeat every 5th retry.
W20260924 12:45:47.007670 5474 catalog_manager.cc:5173] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2e6e1d30fc99418ebeca18a7394a2662 with cas_config_opid_index -1: no extra replica candidate found for tablet 2e6e1d30fc99418ebeca18a7394a2662 (table test-workload [id=8f068027a9be4f6e9e75b857afc47d0f]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
W20260924 12:45:47.037318 5786 generic_service.cc:139] {username='slave'} at 127.0.0.1:33866 forcing change of non-runtime-safe flag fail_dns_resolution
I20260924 12:45:47.037447 5786 generic_service.cc:175] {username='slave'} at 127.0.0.1:33866 changed flags via RPC: fail_dns_resolution from 'true' to 'false'
W20260924 12:45:47.038528 5916 generic_service.cc:139] {username='slave'} at 127.0.0.1:51922 forcing change of non-runtime-safe flag fail_dns_resolution
I20260924 12:45:47.038638 5916 generic_service.cc:175] {username='slave'} at 127.0.0.1:51922 changed flags via RPC: fail_dns_resolution from 'true' to 'false'
W20260924 12:45:47.039618 5655 generic_service.cc:139] {username='slave'} at 127.0.0.1:36542 forcing change of non-runtime-safe flag fail_dns_resolution
I20260924 12:45:47.039727 5655 generic_service.cc:175] {username='slave'} at 127.0.0.1:36542 changed flags via RPC: fail_dns_resolution from 'true' to 'false'
I20260924 12:45:47.100387 5746 raft_consensus.cc:3037] T 2e6e1d30fc99418ebeca18a7394a2662 P 2ffa41fc303343f587ab7d5ef638a311 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:45:47.203097 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.196:0
--local_ip_for_outbound_sockets=127.5.77.196
--webserver_interface=127.5.77.196
--webserver_port=0
--tserver_master_addrs=127.5.77.254:40069
--builtin_ntp_servers=127.5.77.212:41239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--dns_resolver_cache_capacity_mb=0
--enable_leader_failure_detection=false
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=3
--leader_failure_max_missed_heartbeat_periods=10
--leader_failure_exp_backoff_max_delta_ms=5000 with env {}
W20260924 12:45:47.525841 5963 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 12:45:47.526131 5963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:45:47.526209 5963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:45:47.526304 5963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:45:47.533363 5963 flags.cc:425] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=5000
W20260924 12:45:47.533471 5963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:45:47.533631 5963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.196
I20260924 12:45:47.542625 5963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:41239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--enable_leader_failure_detection=false
--leader_failure_exp_backoff_max_delta_ms=5000
--leader_failure_max_missed_heartbeat_periods=10
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.5.77.196
--webserver_port=0
--tserver_master_addrs=127.5.77.254:40069
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--dns_resolver_cache_capacity_mb=0
--local_ip_for_outbound_sockets=127.5.77.196
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:45:47.543892 5963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:45:47.545580 5963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:45:47.557525 5968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:45:47.557547 5971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:45:47.557615 5969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:45:47.558291 5963 server_base.cc:1034] running on GCE node
I20260924 12:45:47.558964 5963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:45:47.559921 5963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:45:47.561158 5963 hybrid_clock.cc:648] HybridClock initialized: now 1790253947561082 us; error 43 us; skew 500 ppm
I20260924 12:45:47.561506 5963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:45:47.567723 5963 webserver.cc:460] Webserver started at http://127.5.77.196:41549/ using document root <none> and password file <none>
I20260924 12:45:47.568435 5963 fs_manager.cc:359] Metadata directory not provided
I20260924 12:45:47.568545 5963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:45:47.568842 5963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:45:47.571504 5963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-3/data/instance:
uuid: "0d40219882224559b23eb9b901016380"
format_stamp: "Formatted at 2026-09-24 12:45:47 on dist-test-slave-b8q3"
I20260924 12:45:47.572263 5963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-3/wal/instance:
uuid: "0d40219882224559b23eb9b901016380"
format_stamp: "Formatted at 2026-09-24 12:45:47 on dist-test-slave-b8q3"
I20260924 12:45:47.577401 5963 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 12:45:47.580804 5977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:47.582178 5963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 12:45:47.582315 5963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-3/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-3/wal
uuid: "0d40219882224559b23eb9b901016380"
format_stamp: "Formatted at 2026-09-24 12:45:47 on dist-test-slave-b8q3"
I20260924 12:45:47.582499 5963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:45:47.613039 5963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:45:47.613879 5963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:45:47.614132 5963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:45:47.615466 5963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:45:47.616871 5963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:45:47.616943 5963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:47.617049 5963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:45:47.617095 5963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:47.647617 5963 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.196:46443
I20260924 12:45:47.647672 6089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.196:46443 every 8 connection(s)
I20260924 12:45:47.648871 5963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-3/data/info.pb
I20260924 12:45:47.658823 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 5963
I20260924 12:45:47.658998 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0/raft_consensus_election-itest-cluster/ts-3/wal/instance
I20260924 12:45:47.663064 6090 heartbeater.cc:347] Connected to a master server at 127.5.77.254:40069
I20260924 12:45:47.663362 6090 heartbeater.cc:464] Registering TS with master...
I20260924 12:45:47.664121 6090 heartbeater.cc:511] Master 127.5.77.254:40069 requested a full tablet report, sending...
I20260924 12:45:47.665400 5474 ts_manager.cc:194] Registered new tserver with Master: 0d40219882224559b23eb9b901016380 (127.5.77.196:46443)
I20260924 12:45:47.666198 5474 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.196:36587
I20260924 12:45:47.676481 5876 consensus_queue.cc:237] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7eee479d28d44f5f842fedc75fde9550" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33047 } } peers { permanent_uuid: "e62a7464cd9147448d8e7a8087cfb6ff" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41341 } } peers { permanent_uuid: "2ffa41fc303343f587ab7d5ef638a311" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 37943 } } peers { permanent_uuid: "0d40219882224559b23eb9b901016380" member_type: NON_VOTER last_known_addr { host: "127.5.77.196" port: 46443 } attrs { promote: true } }
I20260924 12:45:47.679247 5746 raft_consensus.cc:1261] T 2e6e1d30fc99418ebeca18a7394a2662 P 2ffa41fc303343f587ab7d5ef638a311 [term 1 FOLLOWER]: Refusing update from remote peer 7eee479d28d44f5f842fedc75fde9550: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:45:47.679375 5615 raft_consensus.cc:1261] T 2e6e1d30fc99418ebeca18a7394a2662 P e62a7464cd9147448d8e7a8087cfb6ff [term 1 FOLLOWER]: Refusing update from remote peer 7eee479d28d44f5f842fedc75fde9550: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:45:47.680037 5946 consensus_queue.cc:1035] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ffa41fc303343f587ab7d5ef638a311" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 37943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 12:45:47.680387 5956 consensus_queue.cc:1035] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e62a7464cd9147448d8e7a8087cfb6ff" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 12:45:47.682852 5946 raft_consensus.cc:2932] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0d40219882224559b23eb9b901016380 (127.5.77.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7eee479d28d44f5f842fedc75fde9550" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33047 } } peers { permanent_uuid: "e62a7464cd9147448d8e7a8087cfb6ff" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41341 } } peers { permanent_uuid: "2ffa41fc303343f587ab7d5ef638a311" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 37943 } } peers { permanent_uuid: "0d40219882224559b23eb9b901016380" member_type: NON_VOTER last_known_addr { host: "127.5.77.196" port: 46443 } attrs { promote: true } } }
I20260924 12:45:47.683503 5746 raft_consensus.cc:2932] T 2e6e1d30fc99418ebeca18a7394a2662 P 2ffa41fc303343f587ab7d5ef638a311 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0d40219882224559b23eb9b901016380 (127.5.77.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7eee479d28d44f5f842fedc75fde9550" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33047 } } peers { permanent_uuid: "e62a7464cd9147448d8e7a8087cfb6ff" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41341 } } peers { permanent_uuid: "2ffa41fc303343f587ab7d5ef638a311" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 37943 } } peers { permanent_uuid: "0d40219882224559b23eb9b901016380" member_type: NON_VOTER last_known_addr { host: "127.5.77.196" port: 46443 } attrs { promote: true } } }
I20260924 12:45:47.684394 5615 raft_consensus.cc:2932] T 2e6e1d30fc99418ebeca18a7394a2662 P e62a7464cd9147448d8e7a8087cfb6ff [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0d40219882224559b23eb9b901016380 (127.5.77.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7eee479d28d44f5f842fedc75fde9550" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33047 } } peers { permanent_uuid: "e62a7464cd9147448d8e7a8087cfb6ff" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41341 } } peers { permanent_uuid: "2ffa41fc303343f587ab7d5ef638a311" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 37943 } } peers { permanent_uuid: "0d40219882224559b23eb9b901016380" member_type: NON_VOTER last_known_addr { host: "127.5.77.196" port: 46443 } attrs { promote: true } } }
I20260924 12:45:47.689056 5460 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2e6e1d30fc99418ebeca18a7394a2662 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 6)
I20260924 12:45:47.690152 5474 catalog_manager.cc:5495] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 reported cstate change: config changed from index -1 to 2, NON_VOTER 0d40219882224559b23eb9b901016380 (127.5.77.196) added. New cstate: current_term: 1 leader_uuid: "7eee479d28d44f5f842fedc75fde9550" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7eee479d28d44f5f842fedc75fde9550" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e62a7464cd9147448d8e7a8087cfb6ff" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ffa41fc303343f587ab7d5ef638a311" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 37943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d40219882224559b23eb9b901016380" member_type: NON_VOTER last_known_addr { host: "127.5.77.196" port: 46443 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260924 12:45:47.691069 5811 consensus_peers.cc:487] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 -> Peer 0d40219882224559b23eb9b901016380 (127.5.77.196:46443): Couldn't send request to peer 0d40219882224559b23eb9b901016380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e6e1d30fc99418ebeca18a7394a2662. This is attempt 1: this message will repeat every 5th retry.
I20260924 12:45:47.695822 5876 consensus_queue.cc:237] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7eee479d28d44f5f842fedc75fde9550" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33047 } } peers { permanent_uuid: "e62a7464cd9147448d8e7a8087cfb6ff" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41341 } } peers { permanent_uuid: "2ffa41fc303343f587ab7d5ef638a311" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 37943 } }
I20260924 12:45:47.698967 5745 raft_consensus.cc:1261] T 2e6e1d30fc99418ebeca18a7394a2662 P 2ffa41fc303343f587ab7d5ef638a311 [term 1 FOLLOWER]: Refusing update from remote peer 7eee479d28d44f5f842fedc75fde9550: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 12:45:47.699982 5957 consensus_queue.cc:1035] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ffa41fc303343f587ab7d5ef638a311" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 37943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 12:45:47.701049 5614 raft_consensus.cc:1261] T 2e6e1d30fc99418ebeca18a7394a2662 P e62a7464cd9147448d8e7a8087cfb6ff [term 1 FOLLOWER]: Refusing update from remote peer 7eee479d28d44f5f842fedc75fde9550: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 12:45:47.702008 5957 consensus_queue.cc:1035] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e62a7464cd9147448d8e7a8087cfb6ff" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 12:45:47.702543 5957 raft_consensus.cc:2932] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 0d40219882224559b23eb9b901016380 (127.5.77.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7eee479d28d44f5f842fedc75fde9550" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33047 } } peers { permanent_uuid: "e62a7464cd9147448d8e7a8087cfb6ff" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41341 } } peers { permanent_uuid: "2ffa41fc303343f587ab7d5ef638a311" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 37943 } } }
I20260924 12:45:47.703359 5746 raft_consensus.cc:2932] T 2e6e1d30fc99418ebeca18a7394a2662 P 2ffa41fc303343f587ab7d5ef638a311 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 0d40219882224559b23eb9b901016380 (127.5.77.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7eee479d28d44f5f842fedc75fde9550" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33047 } } peers { permanent_uuid: "e62a7464cd9147448d8e7a8087cfb6ff" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41341 } } peers { permanent_uuid: "2ffa41fc303343f587ab7d5ef638a311" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 37943 } } }
I20260924 12:45:47.704851 5614 raft_consensus.cc:2932] T 2e6e1d30fc99418ebeca18a7394a2662 P e62a7464cd9147448d8e7a8087cfb6ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 0d40219882224559b23eb9b901016380 (127.5.77.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7eee479d28d44f5f842fedc75fde9550" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33047 } } peers { permanent_uuid: "e62a7464cd9147448d8e7a8087cfb6ff" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41341 } } peers { permanent_uuid: "2ffa41fc303343f587ab7d5ef638a311" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 37943 } } }
I20260924 12:45:47.706290 5460 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 2e6e1d30fc99418ebeca18a7394a2662 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260924 12:45:47.707343 5474 catalog_manager.cc:5495] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 reported cstate change: config changed from index 2 to 3, NON_VOTER 0d40219882224559b23eb9b901016380 (127.5.77.196) evicted. New cstate: current_term: 1 leader_uuid: "7eee479d28d44f5f842fedc75fde9550" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7eee479d28d44f5f842fedc75fde9550" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e62a7464cd9147448d8e7a8087cfb6ff" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ffa41fc303343f587ab7d5ef638a311" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 37943 } health_report { overall_health: HEALTHY } } }
W20260924 12:45:47.714016 5786 generic_service.cc:139] {username='slave'} at 127.0.0.1:33866 forcing change of non-runtime-safe flag fail_dns_resolution
I20260924 12:45:47.714145 5786 generic_service.cc:175] {username='slave'} at 127.0.0.1:33866 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
W20260924 12:45:47.715315 5916 generic_service.cc:139] {username='slave'} at 127.0.0.1:51922 forcing change of non-runtime-safe flag fail_dns_resolution
I20260924 12:45:47.715436 5916 generic_service.cc:175] {username='slave'} at 127.0.0.1:51922 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
W20260924 12:45:47.717417 5655 generic_service.cc:139] {username='slave'} at 127.0.0.1:36542 forcing change of non-runtime-safe flag fail_dns_resolution
I20260924 12:45:47.717550 5655 generic_service.cc:175] {username='slave'} at 127.0.0.1:36542 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
I20260924 12:45:47.718775 5876 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2e6e1d30fc99418ebeca18a7394a2662"
dest_uuid: "7eee479d28d44f5f842fedc75fde9550"
mode: GRACEFUL
new_leader_uuid: "7eee479d28d44f5f842fedc75fde9550"
from {username='slave'} at 127.0.0.1:51910
I20260924 12:45:47.719058 5876 raft_consensus.cc:595] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 [term 1 LEADER]: Received request to transfer leadership to TS 7eee479d28d44f5f842fedc75fde9550
I20260924 12:45:47.718780 6025 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e6e1d30fc99418ebeca18a7394a2662 with delete_type TABLET_DATA_TOMBSTONED (TS 0d40219882224559b23eb9b901016380 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46166
W20260924 12:45:47.721659 5460 catalog_manager.cc:4808] TS 0d40219882224559b23eb9b901016380 (127.5.77.196:46443): delete failed for tablet 2e6e1d30fc99418ebeca18a7394a2662 because the tablet was not found. No further retry: Not found: Tablet not found: 2e6e1d30fc99418ebeca18a7394a2662
I20260924 12:45:48.668792 6090 heartbeater.cc:503] Master 127.5.77.254:40069 was elected leader, sending a full tablet report...
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/raft_consensus_election-itest.cc:291: Failure
Value of: tablets.empty()
Actual: true
Expected: false
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_util.cc:395: Failure
Failed
Timed out waiting for assertion to pass.
I20260924 12:46:17.807642 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 5533
I20260924 12:46:17.808835 5655 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:17.898888 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 5533
I20260924 12:46:17.913435 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 5664
I20260924 12:46:17.914331 5786 generic_service.cc:196] Checking for leaks (request via RPC)
W20260924 12:46:17.914407 5809 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41341: connect: Connection refused (error 111)
W20260924 12:46:17.915973 5809 consensus_peers.cc:487] T 2e6e1d30fc99418ebeca18a7394a2662 P 7eee479d28d44f5f842fedc75fde9550 -> Peer e62a7464cd9147448d8e7a8087cfb6ff (127.5.77.193:41341): Couldn't send request to peer e62a7464cd9147448d8e7a8087cfb6ff. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41341: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 12:46:18.003129 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 5664
I20260924 12:46:18.022358 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 5794
I20260924 12:46:18.023269 5916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:18.117374 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 5794
I20260924 12:46:18.132190 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 5963
I20260924 12:46:18.138687 6085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:18.221246 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 5963
I20260924 12:46:18.233747 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 5443
I20260924 12:46:18.234865 5504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:18.340242 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 5443
2026-09-24T12:46:18Z chronyd exiting
I20260924 12:46:18.369426 5431 test_util.cc:175] -----------------------------------------------
I20260924 12:46:18.369508 5431 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790253941578571-5431-0
[ FAILED ] RaftConsensusElectionITest.TestNewLeaderCantResolvePeers (36722 ms)
[ RUN ] RaftConsensusElectionITest.RunLeaderElection
I20260924 12:46:18.371554 5431 test_util.cc:268] Using random seed: -59750985
I20260924 12:46:18.373054 5431 ts_itest-base.cc:112] Starting cluster with:
I20260924 12:46:18.373128 5431 ts_itest-base.cc:113] --------------
I20260924 12:46:18.373167 5431 ts_itest-base.cc:114] 3 tablet servers
I20260924 12:46:18.373214 5431 ts_itest-base.cc:115] 3 replicas per TS
I20260924 12:46:18.373247 5431 ts_itest-base.cc:116] --------------
2026-09-24T12:46:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:46:18Z Disabled control of system clock
I20260924 12:46:18.393637 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.77.254:45275
--webserver_interface=127.5.77.254
--webserver_port=0
--builtin_ntp_servers=127.5.77.212:38493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.77.254:45275 with env {}
W20260924 12:46:18.704406 6132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:18.704705 6132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:18.704807 6132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:18.712107 6132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:46:18.712229 6132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:18.712273 6132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:46:18.712348 6132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:46:18.721884 6132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:38493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.77.254:45275
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.77.254:45275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.77.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:18.723229 6132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:18.725003 6132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:18.736701 6137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:18.736709 6140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:18.736735 6138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:18.737478 6132 server_base.cc:1034] running on GCE node
I20260924 12:46:18.738245 6132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:18.739708 6132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:18.740929 6132 hybrid_clock.cc:648] HybridClock initialized: now 1790253978740858 us; error 48 us; skew 500 ppm
I20260924 12:46:18.741256 6132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:46:18.743613 6132 webserver.cc:460] Webserver started at http://127.5.77.254:38191/ using document root <none> and password file <none>
I20260924 12:46:18.744274 6132 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:18.744382 6132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:18.744658 6132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:18.747300 6132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "1c1eec3fb50144ffb2f0ef715802d04f"
format_stamp: "Formatted at 2026-09-24 12:46:18 on dist-test-slave-b8q3"
I20260924 12:46:18.748035 6132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "1c1eec3fb50144ffb2f0ef715802d04f"
format_stamp: "Formatted at 2026-09-24 12:46:18 on dist-test-slave-b8q3"
I20260924 12:46:18.753183 6132 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 12:46:18.756711 6146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:18.758185 6132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 12:46:18.758337 6132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
uuid: "1c1eec3fb50144ffb2f0ef715802d04f"
format_stamp: "Formatted at 2026-09-24 12:46:18 on dist-test-slave-b8q3"
I20260924 12:46:18.758560 6132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:18.777245 6132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:18.778079 6132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:18.778354 6132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:18.795483 6132 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.254:45275
I20260924 12:46:18.795511 6197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.254:45275 every 8 connection(s)
I20260924 12:46:18.796857 6132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260924 12:46:18.797880 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 6132
I20260924 12:46:18.798081 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260924 12:46:18.801628 6198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:18.812223 6198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c1eec3fb50144ffb2f0ef715802d04f: Bootstrap starting.
I20260924 12:46:18.815348 6198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c1eec3fb50144ffb2f0ef715802d04f: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:18.816309 6198 log.cc:824] T 00000000000000000000000000000000 P 1c1eec3fb50144ffb2f0ef715802d04f: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:18.819023 6198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c1eec3fb50144ffb2f0ef715802d04f: No bootstrap required, opened a new log
I20260924 12:46:18.826010 6198 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c1eec3fb50144ffb2f0ef715802d04f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c1eec3fb50144ffb2f0ef715802d04f" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 45275 } }
I20260924 12:46:18.826371 6198 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c1eec3fb50144ffb2f0ef715802d04f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:18.826483 6198 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c1eec3fb50144ffb2f0ef715802d04f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c1eec3fb50144ffb2f0ef715802d04f, State: Initialized, Role: FOLLOWER
I20260924 12:46:18.827064 6198 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c1eec3fb50144ffb2f0ef715802d04f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c1eec3fb50144ffb2f0ef715802d04f" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 45275 } }
I20260924 12:46:18.827270 6198 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c1eec3fb50144ffb2f0ef715802d04f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:46:18.827391 6198 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c1eec3fb50144ffb2f0ef715802d04f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:46:18.827528 6198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c1eec3fb50144ffb2f0ef715802d04f [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:18.829692 6198 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c1eec3fb50144ffb2f0ef715802d04f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c1eec3fb50144ffb2f0ef715802d04f" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 45275 } }
I20260924 12:46:18.830160 6198 leader_election.cc:302] T 00000000000000000000000000000000 P 1c1eec3fb50144ffb2f0ef715802d04f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c1eec3fb50144ffb2f0ef715802d04f; no voters:
I20260924 12:46:18.830731 6198 leader_election.cc:288] T 00000000000000000000000000000000 P 1c1eec3fb50144ffb2f0ef715802d04f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:46:18.830861 6203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c1eec3fb50144ffb2f0ef715802d04f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:18.831506 6203 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c1eec3fb50144ffb2f0ef715802d04f [term 1 LEADER]: Becoming Leader. State: Replica: 1c1eec3fb50144ffb2f0ef715802d04f, State: Running, Role: LEADER
I20260924 12:46:18.832038 6203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c1eec3fb50144ffb2f0ef715802d04f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c1eec3fb50144ffb2f0ef715802d04f" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 45275 } }
I20260924 12:46:18.832621 6198 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c1eec3fb50144ffb2f0ef715802d04f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:46:18.834580 6205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c1eec3fb50144ffb2f0ef715802d04f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c1eec3fb50144ffb2f0ef715802d04f. Latest consensus state: current_term: 1 leader_uuid: "1c1eec3fb50144ffb2f0ef715802d04f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c1eec3fb50144ffb2f0ef715802d04f" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 45275 } } }
I20260924 12:46:18.834905 6205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c1eec3fb50144ffb2f0ef715802d04f [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:18.835341 6204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c1eec3fb50144ffb2f0ef715802d04f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c1eec3fb50144ffb2f0ef715802d04f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c1eec3fb50144ffb2f0ef715802d04f" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 45275 } } }
I20260924 12:46:18.835644 6204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c1eec3fb50144ffb2f0ef715802d04f [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:18.835861 6210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:46:18.841387 6210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:46:18.850538 6210 catalog_manager.cc:1347] Generated new cluster ID: d40f668812e64532b3d6d5ab20b8cb26
I20260924 12:46:18.850714 6210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:46:18.908903 6210 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:46:18.910207 6210 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:46:18.921340 6210 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c1eec3fb50144ffb2f0ef715802d04f: Generated new TSK 0
I20260924 12:46:18.922264 6210 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:46:18.936944 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.193:0
--local_ip_for_outbound_sockets=127.5.77.193
--webserver_interface=127.5.77.193
--webserver_port=0
--tserver_master_addrs=127.5.77.254:45275
--builtin_ntp_servers=127.5.77.212:38493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
W20260924 12:46:19.250972 6222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:19.251257 6222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:19.251377 6222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:19.258558 6222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:19.258754 6222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.193
I20260924 12:46:19.268222 6222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:38493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.77.193
--webserver_port=0
--tserver_master_addrs=127.5.77.254:45275
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.193
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:19.269527 6222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:19.271297 6222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:19.284086 6227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:19.284076 6230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:19.284181 6228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:19.284804 6222 server_base.cc:1034] running on GCE node
I20260924 12:46:19.285378 6222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:19.286316 6222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:19.287549 6222 hybrid_clock.cc:648] HybridClock initialized: now 1790253979287476 us; error 60 us; skew 500 ppm
I20260924 12:46:19.287886 6222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:46:19.290258 6222 webserver.cc:460] Webserver started at http://127.5.77.193:41455/ using document root <none> and password file <none>
I20260924 12:46:19.291019 6222 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:19.291146 6222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:19.291447 6222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:19.294153 6222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "1350a3ec010b436490440535bdff4819"
format_stamp: "Formatted at 2026-09-24 12:46:19 on dist-test-slave-b8q3"
I20260924 12:46:19.294963 6222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "1350a3ec010b436490440535bdff4819"
format_stamp: "Formatted at 2026-09-24 12:46:19 on dist-test-slave-b8q3"
I20260924 12:46:19.300244 6222 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 12:46:19.303794 6236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:19.305130 6222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:46:19.305282 6222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "1350a3ec010b436490440535bdff4819"
format_stamp: "Formatted at 2026-09-24 12:46:19 on dist-test-slave-b8q3"
I20260924 12:46:19.305451 6222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:19.350294 6222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:19.351145 6222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:19.351423 6222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:19.352327 6222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:19.353762 6222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:19.353842 6222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:19.353967 6222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:19.354022 6222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:19.382876 6222 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.193:43027
I20260924 12:46:19.382897 6348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.193:43027 every 8 connection(s)
I20260924 12:46:19.384266 6222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260924 12:46:19.394593 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 6222
I20260924 12:46:19.394748 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260924 12:46:19.397541 6349 heartbeater.cc:347] Connected to a master server at 127.5.77.254:45275
I20260924 12:46:19.397826 6349 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:19.398145 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.194:0
--local_ip_for_outbound_sockets=127.5.77.194
--webserver_interface=127.5.77.194
--webserver_port=0
--tserver_master_addrs=127.5.77.254:45275
--builtin_ntp_servers=127.5.77.212:38493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
I20260924 12:46:19.398591 6349 heartbeater.cc:511] Master 127.5.77.254:45275 requested a full tablet report, sending...
I20260924 12:46:19.400768 6162 ts_manager.cc:194] Registered new tserver with Master: 1350a3ec010b436490440535bdff4819 (127.5.77.193:43027)
I20260924 12:46:19.402585 6162 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.193:46415
W20260924 12:46:19.706049 6353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:19.706348 6353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:19.706485 6353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:19.713775 6353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:19.713975 6353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.194
I20260924 12:46:19.723047 6353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:38493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.77.194
--webserver_port=0
--tserver_master_addrs=127.5.77.254:45275
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.194
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:19.724375 6353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:19.726080 6353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:19.738830 6359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:19.738955 6358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:19.738899 6361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:19.739589 6353 server_base.cc:1034] running on GCE node
I20260924 12:46:19.740149 6353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:19.741060 6353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:19.742264 6353 hybrid_clock.cc:648] HybridClock initialized: now 1790253979742194 us; error 45 us; skew 500 ppm
I20260924 12:46:19.742583 6353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:46:19.750944 6353 webserver.cc:460] Webserver started at http://127.5.77.194:32905/ using document root <none> and password file <none>
I20260924 12:46:19.751677 6353 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:19.751782 6353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:19.752058 6353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:19.754805 6353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "abba09a25a0e4902a42c1dd06291a5d7"
format_stamp: "Formatted at 2026-09-24 12:46:19 on dist-test-slave-b8q3"
I20260924 12:46:19.755546 6353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "abba09a25a0e4902a42c1dd06291a5d7"
format_stamp: "Formatted at 2026-09-24 12:46:19 on dist-test-slave-b8q3"
I20260924 12:46:19.760761 6353 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 12:46:19.764490 6367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:19.765918 6353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 12:46:19.766054 6353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "abba09a25a0e4902a42c1dd06291a5d7"
format_stamp: "Formatted at 2026-09-24 12:46:19 on dist-test-slave-b8q3"
I20260924 12:46:19.766229 6353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:19.804812 6353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:19.805755 6353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:19.806064 6353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:19.807116 6353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:19.808915 6353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:19.809003 6353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:19.809115 6353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:19.809171 6353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:19.839226 6353 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.194:33291
I20260924 12:46:19.839270 6479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.194:33291 every 8 connection(s)
I20260924 12:46:19.840488 6353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260924 12:46:19.845479 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 6353
I20260924 12:46:19.845638 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260924 12:46:19.849305 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.195:0
--local_ip_for_outbound_sockets=127.5.77.195
--webserver_interface=127.5.77.195
--webserver_port=0
--tserver_master_addrs=127.5.77.254:45275
--builtin_ntp_servers=127.5.77.212:38493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
I20260924 12:46:19.855041 6480 heartbeater.cc:347] Connected to a master server at 127.5.77.254:45275
I20260924 12:46:19.855333 6480 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:19.856056 6480 heartbeater.cc:511] Master 127.5.77.254:45275 requested a full tablet report, sending...
I20260924 12:46:19.857431 6162 ts_manager.cc:194] Registered new tserver with Master: abba09a25a0e4902a42c1dd06291a5d7 (127.5.77.194:33291)
I20260924 12:46:19.858201 6162 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.194:60483
W20260924 12:46:20.160678 6483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:20.160964 6483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:20.161039 6483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:20.168426 6483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:20.168599 6483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.195
I20260924 12:46:20.177799 6483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:38493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.77.195
--webserver_port=0
--tserver_master_addrs=127.5.77.254:45275
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.195
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:20.179062 6483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:20.180719 6483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:20.193406 6489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:20.193702 6491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:20.193913 6488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:20.194001 6483 server_base.cc:1034] running on GCE node
I20260924 12:46:20.194728 6483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:20.195688 6483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:20.196911 6483 hybrid_clock.cc:648] HybridClock initialized: now 1790253980196829 us; error 60 us; skew 500 ppm
I20260924 12:46:20.197197 6483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:46:20.199393 6483 webserver.cc:460] Webserver started at http://127.5.77.195:34379/ using document root <none> and password file <none>
I20260924 12:46:20.200063 6483 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:20.200173 6483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:20.200462 6483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:20.203001 6483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "8b9fc28b2190419491e0d7a5fe8afada"
format_stamp: "Formatted at 2026-09-24 12:46:20 on dist-test-slave-b8q3"
I20260924 12:46:20.203740 6483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "8b9fc28b2190419491e0d7a5fe8afada"
format_stamp: "Formatted at 2026-09-24 12:46:20 on dist-test-slave-b8q3"
I20260924 12:46:20.208773 6483 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 12:46:20.212052 6497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:20.213378 6483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 12:46:20.213526 6483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8b9fc28b2190419491e0d7a5fe8afada"
format_stamp: "Formatted at 2026-09-24 12:46:20 on dist-test-slave-b8q3"
I20260924 12:46:20.213708 6483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:20.244704 6483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:20.245519 6483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:20.245828 6483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:20.246753 6483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:20.248131 6483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:20.248206 6483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:20.248327 6483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:20.248378 6483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:20.277081 6483 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.195:43143
I20260924 12:46:20.277136 6609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.195:43143 every 8 connection(s)
I20260924 12:46:20.278332 6483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260924 12:46:20.285617 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 6483
I20260924 12:46:20.285741 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260924 12:46:20.292096 6610 heartbeater.cc:347] Connected to a master server at 127.5.77.254:45275
I20260924 12:46:20.292368 6610 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:20.293054 6610 heartbeater.cc:511] Master 127.5.77.254:45275 requested a full tablet report, sending...
I20260924 12:46:20.294317 6162 ts_manager.cc:194] Registered new tserver with Master: 8b9fc28b2190419491e0d7a5fe8afada (127.5.77.195:43143)
I20260924 12:46:20.295078 6162 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.195:44883
I20260924 12:46:20.301947 5431 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:46:20.317217 6162 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48032:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20260924 12:46:20.324057 6162 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:46:20.356441 6545 tablet_service.cc:1467] Processing CreateTablet for tablet 6f427a34cc694ac094614f876f49ea3b (DEFAULT_TABLE table=TestTable [id=f6bbf4644618489ca7400d3b34e5f106]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:20.358096 6545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f427a34cc694ac094614f876f49ea3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:20.360423 6284 tablet_service.cc:1467] Processing CreateTablet for tablet 6f427a34cc694ac094614f876f49ea3b (DEFAULT_TABLE table=TestTable [id=f6bbf4644618489ca7400d3b34e5f106]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:20.361613 6284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f427a34cc694ac094614f876f49ea3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:20.361704 6415 tablet_service.cc:1467] Processing CreateTablet for tablet 6f427a34cc694ac094614f876f49ea3b (DEFAULT_TABLE table=TestTable [id=f6bbf4644618489ca7400d3b34e5f106]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:20.363025 6415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f427a34cc694ac094614f876f49ea3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:20.372850 6629 tablet_bootstrap.cc:492] T 6f427a34cc694ac094614f876f49ea3b P 1350a3ec010b436490440535bdff4819: Bootstrap starting.
I20260924 12:46:20.373420 6628 tablet_bootstrap.cc:492] T 6f427a34cc694ac094614f876f49ea3b P 8b9fc28b2190419491e0d7a5fe8afada: Bootstrap starting.
I20260924 12:46:20.376262 6628 tablet_bootstrap.cc:654] T 6f427a34cc694ac094614f876f49ea3b P 8b9fc28b2190419491e0d7a5fe8afada: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:20.376412 6629 tablet_bootstrap.cc:654] T 6f427a34cc694ac094614f876f49ea3b P 1350a3ec010b436490440535bdff4819: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:20.377267 6630 tablet_bootstrap.cc:492] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7: Bootstrap starting.
I20260924 12:46:20.377532 6628 log.cc:824] T 6f427a34cc694ac094614f876f49ea3b P 8b9fc28b2190419491e0d7a5fe8afada: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:20.377537 6629 log.cc:824] T 6f427a34cc694ac094614f876f49ea3b P 1350a3ec010b436490440535bdff4819: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:20.380385 6629 tablet_bootstrap.cc:492] T 6f427a34cc694ac094614f876f49ea3b P 1350a3ec010b436490440535bdff4819: No bootstrap required, opened a new log
I20260924 12:46:20.380663 6629 ts_tablet_manager.cc:1397] T 6f427a34cc694ac094614f876f49ea3b P 1350a3ec010b436490440535bdff4819: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260924 12:46:20.380954 6630 tablet_bootstrap.cc:654] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:20.382074 6630 log.cc:824] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:20.383075 6628 tablet_bootstrap.cc:492] T 6f427a34cc694ac094614f876f49ea3b P 8b9fc28b2190419491e0d7a5fe8afada: No bootstrap required, opened a new log
I20260924 12:46:20.383432 6628 ts_tablet_manager.cc:1397] T 6f427a34cc694ac094614f876f49ea3b P 8b9fc28b2190419491e0d7a5fe8afada: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260924 12:46:20.386312 6630 tablet_bootstrap.cc:492] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7: No bootstrap required, opened a new log
I20260924 12:46:20.386629 6630 ts_tablet_manager.cc:1397] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260924 12:46:20.393316 6629 raft_consensus.cc:348] T 6f427a34cc694ac094614f876f49ea3b P 1350a3ec010b436490440535bdff4819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1350a3ec010b436490440535bdff4819" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 43027 } } peers { permanent_uuid: "8b9fc28b2190419491e0d7a5fe8afada" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 43143 } } peers { permanent_uuid: "abba09a25a0e4902a42c1dd06291a5d7" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 33291 } }
I20260924 12:46:20.393811 6629 raft_consensus.cc:374] T 6f427a34cc694ac094614f876f49ea3b P 1350a3ec010b436490440535bdff4819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:20.393921 6629 raft_consensus.cc:729] T 6f427a34cc694ac094614f876f49ea3b P 1350a3ec010b436490440535bdff4819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1350a3ec010b436490440535bdff4819, State: Initialized, Role: FOLLOWER
I20260924 12:46:20.393534 6628 raft_consensus.cc:348] T 6f427a34cc694ac094614f876f49ea3b P 8b9fc28b2190419491e0d7a5fe8afada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1350a3ec010b436490440535bdff4819" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 43027 } } peers { permanent_uuid: "8b9fc28b2190419491e0d7a5fe8afada" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 43143 } } peers { permanent_uuid: "abba09a25a0e4902a42c1dd06291a5d7" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 33291 } }
I20260924 12:46:20.394181 6628 raft_consensus.cc:374] T 6f427a34cc694ac094614f876f49ea3b P 8b9fc28b2190419491e0d7a5fe8afada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:20.394305 6628 raft_consensus.cc:729] T 6f427a34cc694ac094614f876f49ea3b P 8b9fc28b2190419491e0d7a5fe8afada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b9fc28b2190419491e0d7a5fe8afada, State: Initialized, Role: FOLLOWER
I20260924 12:46:20.395112 6628 consensus_queue.cc:260] T 6f427a34cc694ac094614f876f49ea3b P 8b9fc28b2190419491e0d7a5fe8afada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1350a3ec010b436490440535bdff4819" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 43027 } } peers { permanent_uuid: "8b9fc28b2190419491e0d7a5fe8afada" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 43143 } } peers { permanent_uuid: "abba09a25a0e4902a42c1dd06291a5d7" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 33291 } }
I20260924 12:46:20.396008 6629 consensus_queue.cc:260] T 6f427a34cc694ac094614f876f49ea3b P 1350a3ec010b436490440535bdff4819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1350a3ec010b436490440535bdff4819" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 43027 } } peers { permanent_uuid: "8b9fc28b2190419491e0d7a5fe8afada" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 43143 } } peers { permanent_uuid: "abba09a25a0e4902a42c1dd06291a5d7" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 33291 } }
I20260924 12:46:20.396564 6630 raft_consensus.cc:348] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1350a3ec010b436490440535bdff4819" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 43027 } } peers { permanent_uuid: "8b9fc28b2190419491e0d7a5fe8afada" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 43143 } } peers { permanent_uuid: "abba09a25a0e4902a42c1dd06291a5d7" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 33291 } }
I20260924 12:46:20.397066 6630 raft_consensus.cc:374] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:20.397190 6630 raft_consensus.cc:729] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abba09a25a0e4902a42c1dd06291a5d7, State: Initialized, Role: FOLLOWER
I20260924 12:46:20.397801 6630 consensus_queue.cc:260] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1350a3ec010b436490440535bdff4819" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 43027 } } peers { permanent_uuid: "8b9fc28b2190419491e0d7a5fe8afada" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 43143 } } peers { permanent_uuid: "abba09a25a0e4902a42c1dd06291a5d7" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 33291 } }
I20260924 12:46:20.402743 6610 heartbeater.cc:503] Master 127.5.77.254:45275 was elected leader, sending a full tablet report...
I20260924 12:46:20.403863 6628 ts_tablet_manager.cc:1428] T 6f427a34cc694ac094614f876f49ea3b P 8b9fc28b2190419491e0d7a5fe8afada: Time spent starting tablet: real 0.020s user 0.013s sys 0.005s
I20260924 12:46:20.404819 6349 heartbeater.cc:503] Master 127.5.77.254:45275 was elected leader, sending a full tablet report...
I20260924 12:46:20.406008 6629 ts_tablet_manager.cc:1428] T 6f427a34cc694ac094614f876f49ea3b P 1350a3ec010b436490440535bdff4819: Time spent starting tablet: real 0.025s user 0.006s sys 0.014s
I20260924 12:46:20.407773 6480 heartbeater.cc:503] Master 127.5.77.254:45275 was elected leader, sending a full tablet report...
I20260924 12:46:20.408210 6630 ts_tablet_manager.cc:1428] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7: Time spent starting tablet: real 0.021s user 0.013s sys 0.007s
I20260924 12:46:20.418030 6636 raft_consensus.cc:482] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:46:20.418334 6636 raft_consensus.cc:504] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1350a3ec010b436490440535bdff4819" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 43027 } } peers { permanent_uuid: "8b9fc28b2190419491e0d7a5fe8afada" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 43143 } } peers { permanent_uuid: "abba09a25a0e4902a42c1dd06291a5d7" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 33291 } }
I20260924 12:46:20.420106 6636 leader_election.cc:288] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1350a3ec010b436490440535bdff4819 (127.5.77.193:43027), 8b9fc28b2190419491e0d7a5fe8afada (127.5.77.195:43143)
I20260924 12:46:20.426642 6304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f427a34cc694ac094614f876f49ea3b" candidate_uuid: "abba09a25a0e4902a42c1dd06291a5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1350a3ec010b436490440535bdff4819" is_pre_election: true
I20260924 12:46:20.426888 6565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f427a34cc694ac094614f876f49ea3b" candidate_uuid: "abba09a25a0e4902a42c1dd06291a5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b9fc28b2190419491e0d7a5fe8afada" is_pre_election: true
I20260924 12:46:20.427134 6304 raft_consensus.cc:2454] T 6f427a34cc694ac094614f876f49ea3b P 1350a3ec010b436490440535bdff4819 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abba09a25a0e4902a42c1dd06291a5d7 in term 0.
I20260924 12:46:20.427384 6565 raft_consensus.cc:2454] T 6f427a34cc694ac094614f876f49ea3b P 8b9fc28b2190419491e0d7a5fe8afada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abba09a25a0e4902a42c1dd06291a5d7 in term 0.
I20260924 12:46:20.427755 6370 leader_election.cc:302] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1350a3ec010b436490440535bdff4819, abba09a25a0e4902a42c1dd06291a5d7; no voters:
I20260924 12:46:20.428161 6636 raft_consensus.cc:2781] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:46:20.428287 6636 raft_consensus.cc:482] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:46:20.428373 6636 raft_consensus.cc:3037] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:20.430333 6636 raft_consensus.cc:504] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1350a3ec010b436490440535bdff4819" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 43027 } } peers { permanent_uuid: "8b9fc28b2190419491e0d7a5fe8afada" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 43143 } } peers { permanent_uuid: "abba09a25a0e4902a42c1dd06291a5d7" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 33291 } }
I20260924 12:46:20.430925 6636 leader_election.cc:288] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7 [CANDIDATE]: Term 1 election: Requested vote from peers 1350a3ec010b436490440535bdff4819 (127.5.77.193:43027), 8b9fc28b2190419491e0d7a5fe8afada (127.5.77.195:43143)
I20260924 12:46:20.431330 6304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f427a34cc694ac094614f876f49ea3b" candidate_uuid: "abba09a25a0e4902a42c1dd06291a5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1350a3ec010b436490440535bdff4819"
I20260924 12:46:20.431406 6565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f427a34cc694ac094614f876f49ea3b" candidate_uuid: "abba09a25a0e4902a42c1dd06291a5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b9fc28b2190419491e0d7a5fe8afada"
I20260924 12:46:20.431546 6304 raft_consensus.cc:3037] T 6f427a34cc694ac094614f876f49ea3b P 1350a3ec010b436490440535bdff4819 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:20.431571 6565 raft_consensus.cc:3037] T 6f427a34cc694ac094614f876f49ea3b P 8b9fc28b2190419491e0d7a5fe8afada [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:20.434501 6304 raft_consensus.cc:2454] T 6f427a34cc694ac094614f876f49ea3b P 1350a3ec010b436490440535bdff4819 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abba09a25a0e4902a42c1dd06291a5d7 in term 1.
I20260924 12:46:20.434517 6565 raft_consensus.cc:2454] T 6f427a34cc694ac094614f876f49ea3b P 8b9fc28b2190419491e0d7a5fe8afada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abba09a25a0e4902a42c1dd06291a5d7 in term 1.
I20260924 12:46:20.434919 6370 leader_election.cc:302] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1350a3ec010b436490440535bdff4819, abba09a25a0e4902a42c1dd06291a5d7; no voters:
I20260924 12:46:20.435240 6636 raft_consensus.cc:2781] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:20.435707 6636 raft_consensus.cc:686] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7 [term 1 LEADER]: Becoming Leader. State: Replica: abba09a25a0e4902a42c1dd06291a5d7, State: Running, Role: LEADER
I20260924 12:46:20.436179 6636 consensus_queue.cc:237] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1350a3ec010b436490440535bdff4819" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 43027 } } peers { permanent_uuid: "8b9fc28b2190419491e0d7a5fe8afada" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 43143 } } peers { permanent_uuid: "abba09a25a0e4902a42c1dd06291a5d7" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 33291 } }
I20260924 12:46:20.440747 6162 catalog_manager.cc:5495] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to abba09a25a0e4902a42c1dd06291a5d7 (127.5.77.194). New cstate: current_term: 1 leader_uuid: "abba09a25a0e4902a42c1dd06291a5d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1350a3ec010b436490440535bdff4819" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 43027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b9fc28b2190419491e0d7a5fe8afada" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 43143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abba09a25a0e4902a42c1dd06291a5d7" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 33291 } health_report { overall_health: HEALTHY } } }
I20260924 12:46:20.461329 5431 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:46:20.464664 5431 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1350a3ec010b436490440535bdff4819 to finish bootstrapping
I20260924 12:46:20.473243 5431 ts_itest-base.cc:233] Waiting for 1 tablets on tserver abba09a25a0e4902a42c1dd06291a5d7 to finish bootstrapping
I20260924 12:46:20.481213 5431 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 8b9fc28b2190419491e0d7a5fe8afada to finish bootstrapping
I20260924 12:46:20.517982 6565 raft_consensus.cc:1261] T 6f427a34cc694ac094614f876f49ea3b P 8b9fc28b2190419491e0d7a5fe8afada [term 1 FOLLOWER]: Refusing update from remote peer abba09a25a0e4902a42c1dd06291a5d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:20.518036 6304 raft_consensus.cc:1261] T 6f427a34cc694ac094614f876f49ea3b P 1350a3ec010b436490440535bdff4819 [term 1 FOLLOWER]: Refusing update from remote peer abba09a25a0e4902a42c1dd06291a5d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:20.518942 6639 consensus_queue.cc:1035] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9fc28b2190419491e0d7a5fe8afada" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 43143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:20.519294 6636 consensus_queue.cc:1035] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1350a3ec010b436490440535bdff4819" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 43027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 12:46:20.529860 6611 tablet.cc:2369] T 6f427a34cc694ac094614f876f49ea3b P 8b9fc28b2190419491e0d7a5fe8afada: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:46:20.536810 6645 mvcc.cc:205] Tried to move back new op lower bound from 7332880304191381504 to 7332880303870377984. Current Snapshot: MvccSnapshot[applied={T|T < 7332880304191381504}]
I20260924 12:46:20.684584 6284 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260924 12:46:20.685029 6545 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260924 12:46:20.685933 6415 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
1c1eec3fb50144ffb2f0ef715802d04f | 127.5.77.254:45275 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.77.212:38493 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
1350a3ec010b436490440535bdff4819 | 127.5.77.193:43027 | HEALTHY | <none> | 0 | 0
8b9fc28b2190419491e0d7a5fe8afada | 127.5.77.195:43143 | HEALTHY | <none> | 0 | 0
abba09a25a0e4902a42c1dd06291a5d7 | 127.5.77.194:33291 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.5.77.193 | experimental | 127.5.77.193:43027
local_ip_for_outbound_sockets | 127.5.77.194 | experimental | 127.5.77.194:33291
local_ip_for_outbound_sockets | 127.5.77.195 | experimental | 127.5.77.195:43143
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.5.77.193:43027
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.77.194:33291
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.77.195:43143
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.77.212:38493 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260924 12:46:20.730710 5431 log_verifier.cc:126] Checking tablet 6f427a34cc694ac094614f876f49ea3b
I20260924 12:46:20.768862 5431 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 6f427a34cc694ac094614f876f49ea3b
I20260924 12:46:20.826874 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 6353
I20260924 12:46:20.827708 6475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:20.928290 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 6353
I20260924 12:46:20.942447 6303 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6f427a34cc694ac094614f876f49ea3b"
dest_uuid: "1350a3ec010b436490440535bdff4819"
from {username='slave'} at 127.0.0.1:47770
I20260924 12:46:20.942787 6303 raft_consensus.cc:482] T 6f427a34cc694ac094614f876f49ea3b P 1350a3ec010b436490440535bdff4819 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 12:46:20.942942 6303 raft_consensus.cc:3037] T 6f427a34cc694ac094614f876f49ea3b P 1350a3ec010b436490440535bdff4819 [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:46:20.945596 6303 raft_consensus.cc:504] T 6f427a34cc694ac094614f876f49ea3b P 1350a3ec010b436490440535bdff4819 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1350a3ec010b436490440535bdff4819" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 43027 } } peers { permanent_uuid: "8b9fc28b2190419491e0d7a5fe8afada" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 43143 } } peers { permanent_uuid: "abba09a25a0e4902a42c1dd06291a5d7" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 33291 } }
I20260924 12:46:20.947009 6303 leader_election.cc:288] T 6f427a34cc694ac094614f876f49ea3b P 1350a3ec010b436490440535bdff4819 [CANDIDATE]: Term 2 election: Requested vote from peers 8b9fc28b2190419491e0d7a5fe8afada (127.5.77.195:43143), abba09a25a0e4902a42c1dd06291a5d7 (127.5.77.194:33291)
W20260924 12:46:20.954963 6240 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.77.194:33291: connect: Connection refused (error 111)
I20260924 12:46:20.956234 6564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f427a34cc694ac094614f876f49ea3b" candidate_uuid: "1350a3ec010b436490440535bdff4819" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "8b9fc28b2190419491e0d7a5fe8afada"
I20260924 12:46:20.956485 6564 raft_consensus.cc:3037] T 6f427a34cc694ac094614f876f49ea3b P 8b9fc28b2190419491e0d7a5fe8afada [term 1 FOLLOWER]: Advancing to term 2
W20260924 12:46:20.956632 6240 leader_election.cc:334] T 6f427a34cc694ac094614f876f49ea3b P 1350a3ec010b436490440535bdff4819 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer abba09a25a0e4902a42c1dd06291a5d7 (127.5.77.194:33291): Network error: Client connection negotiation failed: client connection to 127.5.77.194:33291: connect: Connection refused (error 111)
W20260924 12:46:20.956691 6616 meta_cache.cc:302] tablet 6f427a34cc694ac094614f876f49ea3b: replica abba09a25a0e4902a42c1dd06291a5d7 (127.5.77.194:33291) has failed: Network error: Client connection negotiation failed: client connection to 127.5.77.194:33291: connect: Connection refused (error 111)
I20260924 12:46:20.958727 6564 raft_consensus.cc:2454] T 6f427a34cc694ac094614f876f49ea3b P 8b9fc28b2190419491e0d7a5fe8afada [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1350a3ec010b436490440535bdff4819 in term 2.
I20260924 12:46:20.959224 6240 leader_election.cc:302] T 6f427a34cc694ac094614f876f49ea3b P 1350a3ec010b436490440535bdff4819 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1350a3ec010b436490440535bdff4819, 8b9fc28b2190419491e0d7a5fe8afada; no voters: abba09a25a0e4902a42c1dd06291a5d7
I20260924 12:46:20.959582 6635 raft_consensus.cc:2781] T 6f427a34cc694ac094614f876f49ea3b P 1350a3ec010b436490440535bdff4819 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 12:46:20.960115 6635 raft_consensus.cc:686] T 6f427a34cc694ac094614f876f49ea3b P 1350a3ec010b436490440535bdff4819 [term 2 LEADER]: Becoming Leader. State: Replica: 1350a3ec010b436490440535bdff4819, State: Running, Role: LEADER
I20260924 12:46:20.960584 6635 consensus_queue.cc:237] T 6f427a34cc694ac094614f876f49ea3b P 1350a3ec010b436490440535bdff4819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1350a3ec010b436490440535bdff4819" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 43027 } } peers { permanent_uuid: "8b9fc28b2190419491e0d7a5fe8afada" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 43143 } } peers { permanent_uuid: "abba09a25a0e4902a42c1dd06291a5d7" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 33291 } }
I20260924 12:46:20.963660 6160 catalog_manager.cc:5495] T 6f427a34cc694ac094614f876f49ea3b P 1350a3ec010b436490440535bdff4819 reported cstate change: term changed from 1 to 2, leader changed from abba09a25a0e4902a42c1dd06291a5d7 (127.5.77.194) to 1350a3ec010b436490440535bdff4819 (127.5.77.193). New cstate: current_term: 2 leader_uuid: "1350a3ec010b436490440535bdff4819" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1350a3ec010b436490440535bdff4819" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 43027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b9fc28b2190419491e0d7a5fe8afada" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 43143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abba09a25a0e4902a42c1dd06291a5d7" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 33291 } health_report { overall_health: UNKNOWN } } }
I20260924 12:46:20.973008 6564 raft_consensus.cc:1261] T 6f427a34cc694ac094614f876f49ea3b P 8b9fc28b2190419491e0d7a5fe8afada [term 2 FOLLOWER]: Refusing update from remote peer 1350a3ec010b436490440535bdff4819: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20260924 12:46:20.974689 6635 consensus_queue.cc:1035] T 6f427a34cc694ac094614f876f49ea3b P 1350a3ec010b436490440535bdff4819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9fc28b2190419491e0d7a5fe8afada" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 43143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260924 12:46:20.975029 6240 consensus_peers.cc:487] T 6f427a34cc694ac094614f876f49ea3b P 1350a3ec010b436490440535bdff4819 -> Peer abba09a25a0e4902a42c1dd06291a5d7 (127.5.77.194:33291): Couldn't send request to peer abba09a25a0e4902a42c1dd06291a5d7. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.194:33291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 12:46:20.982723 6644 mvcc.cc:205] Tried to move back new op lower bound from 7332880306058145792 to 7332880306018209792. Current Snapshot: MvccSnapshot[applied={T|T < 7332880306058145792 or (T in {7332880306058145792})}]
I20260924 12:46:21.034039 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.194:33291
--local_ip_for_outbound_sockets=127.5.77.194
--tserver_master_addrs=127.5.77.254:45275
--webserver_port=32905
--webserver_interface=127.5.77.194
--builtin_ntp_servers=127.5.77.212:38493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
W20260924 12:46:21.354707 6691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:21.354986 6691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:21.355100 6691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:21.362691 6691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:21.362874 6691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.194
I20260924 12:46:21.372109 6691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:38493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.194:33291
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.77.194
--webserver_port=32905
--tserver_master_addrs=127.5.77.254:45275
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.194
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:21.373329 6691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:21.375070 6691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:21.387441 6696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:21.387568 6697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:21.387454 6699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:21.389109 6691 server_base.cc:1034] running on GCE node
I20260924 12:46:21.389781 6691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:21.390857 6691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:21.392098 6691 hybrid_clock.cc:648] HybridClock initialized: now 1790253981392033 us; error 43 us; skew 500 ppm
I20260924 12:46:21.392398 6691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:46:21.394812 6691 webserver.cc:460] Webserver started at http://127.5.77.194:32905/ using document root <none> and password file <none>
I20260924 12:46:21.395464 6691 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:21.395581 6691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:21.400593 6691 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260924 12:46:21.403426 6705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:21.404757 6691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 12:46:21.404919 6691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "abba09a25a0e4902a42c1dd06291a5d7"
format_stamp: "Formatted at 2026-09-24 12:46:19 on dist-test-slave-b8q3"
I20260924 12:46:21.405846 6691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:21.422155 6691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:21.423018 6691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:21.423272 6691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:21.424256 6691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:21.426064 6712 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260924 12:46:21.431478 6691 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 12:46:21.431581 6691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260924 12:46:21.431700 6691 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 12:46:21.434712 6691 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 12:46:21.434782 6691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260924 12:46:21.435096 6712 tablet_bootstrap.cc:492] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7: Bootstrap starting.
I20260924 12:46:21.457099 6712 log.cc:824] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:21.471872 6691 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.194:33291
I20260924 12:46:21.471903 6819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.194:33291 every 8 connection(s)
I20260924 12:46:21.473045 6691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260924 12:46:21.480392 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 6691
I20260924 12:46:21.488543 6820 heartbeater.cc:347] Connected to a master server at 127.5.77.254:45275
I20260924 12:46:21.488900 6820 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:21.489660 6820 heartbeater.cc:511] Master 127.5.77.254:45275 requested a full tablet report, sending...
I20260924 12:46:21.492350 6161 ts_manager.cc:194] Re-registered known tserver with Master: abba09a25a0e4902a42c1dd06291a5d7 (127.5.77.194:33291)
I20260924 12:46:21.493577 6161 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.194:40205
I20260924 12:46:21.495899 6712 tablet_bootstrap.cc:492] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=500 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 12:46:21.496418 6712 tablet_bootstrap.cc:492] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7: Bootstrap complete.
I20260924 12:46:21.497361 6712 ts_tablet_manager.cc:1397] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7: Time spent bootstrapping tablet: real 0.063s user 0.054s sys 0.005s
I20260924 12:46:21.502211 6712 raft_consensus.cc:348] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1350a3ec010b436490440535bdff4819" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 43027 } } peers { permanent_uuid: "8b9fc28b2190419491e0d7a5fe8afada" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 43143 } } peers { permanent_uuid: "abba09a25a0e4902a42c1dd06291a5d7" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 33291 } }
I20260924 12:46:21.503118 6712 raft_consensus.cc:729] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: abba09a25a0e4902a42c1dd06291a5d7, State: Initialized, Role: FOLLOWER
I20260924 12:46:21.503669 6712 consensus_queue.cc:260] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1350a3ec010b436490440535bdff4819" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 43027 } } peers { permanent_uuid: "8b9fc28b2190419491e0d7a5fe8afada" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 43143 } } peers { permanent_uuid: "abba09a25a0e4902a42c1dd06291a5d7" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 33291 } }
I20260924 12:46:21.504627 6820 heartbeater.cc:503] Master 127.5.77.254:45275 was elected leader, sending a full tablet report...
I20260924 12:46:21.505299 6712 ts_tablet_manager.cc:1428] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7: Time spent starting tablet: real 0.008s user 0.011s sys 0.000s
I20260924 12:46:21.523036 6774 raft_consensus.cc:3037] T 6f427a34cc694ac094614f876f49ea3b P abba09a25a0e4902a42c1dd06291a5d7 [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:46:21.540431 6839 mvcc.cc:205] Tried to move back new op lower bound from 7332880306058145792 to 7332880306018209792. Current Snapshot: MvccSnapshot[applied={T|T < 7332880306058145792}]
I20260924 12:46:21.574875 6284 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260924 12:46:21.577128 6754 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260924 12:46:21.581761 6545 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
1c1eec3fb50144ffb2f0ef715802d04f | 127.5.77.254:45275 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.77.212:38493 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
1350a3ec010b436490440535bdff4819 | 127.5.77.193:43027 | HEALTHY | <none> | 1 | 0
8b9fc28b2190419491e0d7a5fe8afada | 127.5.77.195:43143 | HEALTHY | <none> | 0 | 0
abba09a25a0e4902a42c1dd06291a5d7 | 127.5.77.194:33291 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.5.77.193 | experimental | 127.5.77.193:43027
local_ip_for_outbound_sockets | 127.5.77.194 | experimental | 127.5.77.194:33291
local_ip_for_outbound_sockets | 127.5.77.195 | experimental | 127.5.77.195:43143
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.5.77.193:43027
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.77.194:33291
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.77.195:43143
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.77.212:38493 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260924 12:46:21.620072 5431 log_verifier.cc:126] Checking tablet 6f427a34cc694ac094614f876f49ea3b
I20260924 12:46:21.661499 5431 log_verifier.cc:177] Verified matching terms for 12 ops in tablet 6f427a34cc694ac094614f876f49ea3b
I20260924 12:46:22.035480 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 6222
I20260924 12:46:22.037159 6344 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:22.130266 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 6222
I20260924 12:46:22.150345 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 6691
I20260924 12:46:22.157359 6815 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:22.261155 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 6691
I20260924 12:46:22.274492 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 6483
I20260924 12:46:22.275422 6605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:22.365168 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 6483
I20260924 12:46:22.377727 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 6132
I20260924 12:46:22.378718 6193 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:22.487773 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 6132
2026-09-24T12:46:22Z chronyd exiting
[ OK ] RaftConsensusElectionITest.RunLeaderElection (4153 ms)
[ RUN ] RaftConsensusElectionITest.ChurnyElections
I20260924 12:46:22.524633 5431 test_util.cc:268] Using random seed: -55597884
I20260924 12:46:22.526235 5431 ts_itest-base.cc:112] Starting cluster with:
I20260924 12:46:22.526294 5431 ts_itest-base.cc:113] --------------
I20260924 12:46:22.526333 5431 ts_itest-base.cc:114] 3 tablet servers
I20260924 12:46:22.526366 5431 ts_itest-base.cc:115] 3 replicas per TS
I20260924 12:46:22.526413 5431 ts_itest-base.cc:116] --------------
2026-09-24T12:46:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:46:22Z Disabled control of system clock
I20260924 12:46:22.548198 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.77.254:46369
--webserver_interface=127.5.77.254
--webserver_port=0
--builtin_ntp_servers=127.5.77.212:45959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.77.254:46369 with env {}
W20260924 12:46:22.857537 6880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:22.857831 6880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:22.857986 6880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:22.865186 6880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:46:22.865300 6880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:22.865341 6880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:46:22.865414 6880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:46:22.875007 6880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:45959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.77.254:46369
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.77.254:46369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.77.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:22.876700 6880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:22.878928 6880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:22.890938 6886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:22.890949 6885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:22.890954 6888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:22.891743 6880 server_base.cc:1034] running on GCE node
I20260924 12:46:22.892506 6880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:22.894100 6880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:22.895299 6880 hybrid_clock.cc:648] HybridClock initialized: now 1790253982895237 us; error 43 us; skew 500 ppm
I20260924 12:46:22.895609 6880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:46:22.897753 6880 webserver.cc:460] Webserver started at http://127.5.77.254:46297/ using document root <none> and password file <none>
I20260924 12:46:22.898391 6880 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:22.898533 6880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:22.898813 6880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:22.901350 6880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d0910ced495e438f8b398033c51ef6f5"
format_stamp: "Formatted at 2026-09-24 12:46:22 on dist-test-slave-b8q3"
I20260924 12:46:22.902060 6880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d0910ced495e438f8b398033c51ef6f5"
format_stamp: "Formatted at 2026-09-24 12:46:22 on dist-test-slave-b8q3"
I20260924 12:46:22.907099 6880 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 12:46:22.910351 6894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:22.911650 6880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:46:22.911801 6880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d0910ced495e438f8b398033c51ef6f5"
format_stamp: "Formatted at 2026-09-24 12:46:22 on dist-test-slave-b8q3"
I20260924 12:46:22.911994 6880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:22.926347 6880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:22.927174 6880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:22.927457 6880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:22.943958 6880 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.254:46369
I20260924 12:46:22.943982 6945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.254:46369 every 8 connection(s)
I20260924 12:46:22.945242 6880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260924 12:46:22.948925 6946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:22.953998 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 6880
I20260924 12:46:22.954154 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260924 12:46:22.959103 6946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0910ced495e438f8b398033c51ef6f5: Bootstrap starting.
I20260924 12:46:22.963193 6946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0910ced495e438f8b398033c51ef6f5: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:22.964411 6946 log.cc:824] T 00000000000000000000000000000000 P d0910ced495e438f8b398033c51ef6f5: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:22.967420 6946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0910ced495e438f8b398033c51ef6f5: No bootstrap required, opened a new log
I20260924 12:46:22.975044 6946 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0910ced495e438f8b398033c51ef6f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0910ced495e438f8b398033c51ef6f5" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 46369 } }
I20260924 12:46:22.975384 6946 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0910ced495e438f8b398033c51ef6f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:22.975486 6946 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0910ced495e438f8b398033c51ef6f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0910ced495e438f8b398033c51ef6f5, State: Initialized, Role: FOLLOWER
I20260924 12:46:22.976002 6946 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0910ced495e438f8b398033c51ef6f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0910ced495e438f8b398033c51ef6f5" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 46369 } }
I20260924 12:46:22.976222 6946 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0910ced495e438f8b398033c51ef6f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:46:22.976346 6946 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0910ced495e438f8b398033c51ef6f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:46:22.976485 6946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0910ced495e438f8b398033c51ef6f5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:22.978763 6946 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0910ced495e438f8b398033c51ef6f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0910ced495e438f8b398033c51ef6f5" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 46369 } }
I20260924 12:46:22.979228 6946 leader_election.cc:302] T 00000000000000000000000000000000 P d0910ced495e438f8b398033c51ef6f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0910ced495e438f8b398033c51ef6f5; no voters:
I20260924 12:46:22.979758 6946 leader_election.cc:288] T 00000000000000000000000000000000 P d0910ced495e438f8b398033c51ef6f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:46:22.979874 6951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0910ced495e438f8b398033c51ef6f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:22.980458 6951 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0910ced495e438f8b398033c51ef6f5 [term 1 LEADER]: Becoming Leader. State: Replica: d0910ced495e438f8b398033c51ef6f5, State: Running, Role: LEADER
I20260924 12:46:22.980932 6951 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0910ced495e438f8b398033c51ef6f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0910ced495e438f8b398033c51ef6f5" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 46369 } }
I20260924 12:46:22.981648 6946 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0910ced495e438f8b398033c51ef6f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:46:22.983291 6952 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0910ced495e438f8b398033c51ef6f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0910ced495e438f8b398033c51ef6f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0910ced495e438f8b398033c51ef6f5" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 46369 } } }
I20260924 12:46:22.983651 6952 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0910ced495e438f8b398033c51ef6f5 [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:22.984409 6953 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0910ced495e438f8b398033c51ef6f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0910ced495e438f8b398033c51ef6f5. Latest consensus state: current_term: 1 leader_uuid: "d0910ced495e438f8b398033c51ef6f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0910ced495e438f8b398033c51ef6f5" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 46369 } } }
I20260924 12:46:22.984740 6953 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0910ced495e438f8b398033c51ef6f5 [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:22.987213 6956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:46:22.991931 6956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:46:23.000041 6956 catalog_manager.cc:1347] Generated new cluster ID: fc411aee79814f629ac6c1a4dffbd0e6
I20260924 12:46:23.000144 6956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:46:23.028666 6956 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:46:23.029974 6956 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:46:23.039080 6956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0910ced495e438f8b398033c51ef6f5: Generated new TSK 0
I20260924 12:46:23.039979 6956 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:46:23.049314 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.193:0
--local_ip_for_outbound_sockets=127.5.77.193
--webserver_interface=127.5.77.193
--webserver_port=0
--tserver_master_addrs=127.5.77.254:46369
--builtin_ntp_servers=127.5.77.212:45959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
W20260924 12:46:23.363287 6970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:23.363597 6970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:23.363709 6970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:23.371292 6970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:23.371488 6970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.193
I20260924 12:46:23.380546 6970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:45959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.77.193
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.5.77.254:46369
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.193
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:23.381848 6970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:23.383662 6970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:23.396415 6976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:23.396441 6978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:23.396632 6975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:23.397300 6970 server_base.cc:1034] running on GCE node
I20260924 12:46:23.397965 6970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:23.398938 6970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:23.400156 6970 hybrid_clock.cc:648] HybridClock initialized: now 1790253983400083 us; error 53 us; skew 500 ppm
I20260924 12:46:23.400444 6970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:46:23.402736 6970 webserver.cc:460] Webserver started at http://127.5.77.193:36607/ using document root <none> and password file <none>
I20260924 12:46:23.403445 6970 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:23.403561 6970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:23.403847 6970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:23.408134 6970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "9db3b3e723eb41299037b5eab1b00fde"
format_stamp: "Formatted at 2026-09-24 12:46:23 on dist-test-slave-b8q3"
I20260924 12:46:23.408885 6970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "9db3b3e723eb41299037b5eab1b00fde"
format_stamp: "Formatted at 2026-09-24 12:46:23 on dist-test-slave-b8q3"
I20260924 12:46:23.414095 6970 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 12:46:23.417694 6984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:23.419080 6970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:46:23.419226 6970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "9db3b3e723eb41299037b5eab1b00fde"
format_stamp: "Formatted at 2026-09-24 12:46:23 on dist-test-slave-b8q3"
I20260924 12:46:23.419395 6970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:23.437124 6970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:23.437959 6970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:23.438230 6970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:23.439201 6970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:23.440629 6970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:23.440706 6970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:23.440826 6970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:23.440877 6970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:23.470625 6970 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.193:36435
I20260924 12:46:23.470667 7096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.193:36435 every 8 connection(s)
I20260924 12:46:23.471958 6970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260924 12:46:23.477157 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 6970
I20260924 12:46:23.477327 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260924 12:46:23.480537 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.194:0
--local_ip_for_outbound_sockets=127.5.77.194
--webserver_interface=127.5.77.194
--webserver_port=0
--tserver_master_addrs=127.5.77.254:46369
--builtin_ntp_servers=127.5.77.212:45959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
I20260924 12:46:23.485605 7097 heartbeater.cc:347] Connected to a master server at 127.5.77.254:46369
I20260924 12:46:23.485884 7097 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:23.486725 7097 heartbeater.cc:511] Master 127.5.77.254:46369 requested a full tablet report, sending...
I20260924 12:46:23.488767 6911 ts_manager.cc:194] Registered new tserver with Master: 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435)
I20260924 12:46:23.490235 6911 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.193:38885
W20260924 12:46:23.788654 7101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:23.788993 7101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:23.789124 7101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:23.796403 7101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:23.796602 7101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.194
I20260924 12:46:23.805639 7101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:45959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.77.194
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.5.77.254:46369
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.194
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:23.806974 7101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:23.808684 7101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:23.822497 7107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:23.822647 7106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:23.822508 7109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:23.823118 7101 server_base.cc:1034] running on GCE node
I20260924 12:46:23.823756 7101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:23.824642 7101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:23.825851 7101 hybrid_clock.cc:648] HybridClock initialized: now 1790253983825776 us; error 47 us; skew 500 ppm
I20260924 12:46:23.826161 7101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:46:23.828260 7101 webserver.cc:460] Webserver started at http://127.5.77.194:43703/ using document root <none> and password file <none>
I20260924 12:46:23.828886 7101 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:23.828984 7101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:23.829241 7101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:23.831768 7101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "6fb4103951824ad892bfc6e460c4069d"
format_stamp: "Formatted at 2026-09-24 12:46:23 on dist-test-slave-b8q3"
I20260924 12:46:23.832432 7101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "6fb4103951824ad892bfc6e460c4069d"
format_stamp: "Formatted at 2026-09-24 12:46:23 on dist-test-slave-b8q3"
I20260924 12:46:23.837384 7101 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 12:46:23.840939 7115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:23.842305 7101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 12:46:23.842453 7101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "6fb4103951824ad892bfc6e460c4069d"
format_stamp: "Formatted at 2026-09-24 12:46:23 on dist-test-slave-b8q3"
I20260924 12:46:23.842648 7101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:23.857003 7101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:23.857827 7101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:23.858070 7101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:23.858994 7101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:23.860363 7101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:23.860433 7101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:23.860520 7101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:23.860561 7101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:23.889433 7101 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.194:35473
I20260924 12:46:23.889469 7227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.194:35473 every 8 connection(s)
I20260924 12:46:23.890671 7101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260924 12:46:23.897691 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 7101
I20260924 12:46:23.897859 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260924 12:46:23.901008 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.195:0
--local_ip_for_outbound_sockets=127.5.77.195
--webserver_interface=127.5.77.195
--webserver_port=0
--tserver_master_addrs=127.5.77.254:46369
--builtin_ntp_servers=127.5.77.212:45959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
I20260924 12:46:23.904390 7228 heartbeater.cc:347] Connected to a master server at 127.5.77.254:46369
I20260924 12:46:23.904722 7228 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:23.905524 7228 heartbeater.cc:511] Master 127.5.77.254:46369 requested a full tablet report, sending...
I20260924 12:46:23.906855 6911 ts_manager.cc:194] Registered new tserver with Master: 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473)
I20260924 12:46:23.907598 6911 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.194:56209
W20260924 12:46:24.215144 7231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:24.215427 7231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:24.215536 7231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:24.222976 7231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:24.223173 7231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.195
I20260924 12:46:24.232409 7231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:45959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.77.195
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.5.77.254:46369
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.195
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:24.233651 7231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:24.235541 7231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:24.248584 7237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:24.248508 7236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:24.248558 7239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:24.249388 7231 server_base.cc:1034] running on GCE node
I20260924 12:46:24.250015 7231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:24.251005 7231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:24.252226 7231 hybrid_clock.cc:648] HybridClock initialized: now 1790253984252163 us; error 38 us; skew 500 ppm
I20260924 12:46:24.252523 7231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:46:24.254827 7231 webserver.cc:460] Webserver started at http://127.5.77.195:35517/ using document root <none> and password file <none>
I20260924 12:46:24.255507 7231 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:24.255623 7231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:24.255913 7231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:24.258486 7231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "7632672814be4a688ec7ab18a503a58e"
format_stamp: "Formatted at 2026-09-24 12:46:24 on dist-test-slave-b8q3"
I20260924 12:46:24.259241 7231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "7632672814be4a688ec7ab18a503a58e"
format_stamp: "Formatted at 2026-09-24 12:46:24 on dist-test-slave-b8q3"
I20260924 12:46:24.264225 7231 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260924 12:46:24.267510 7245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:24.268841 7231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 12:46:24.269001 7231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7632672814be4a688ec7ab18a503a58e"
format_stamp: "Formatted at 2026-09-24 12:46:24 on dist-test-slave-b8q3"
I20260924 12:46:24.269165 7231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:24.291790 7231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:24.292661 7231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:24.292932 7231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:24.293808 7231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:24.295248 7231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:24.295321 7231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:24.295435 7231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:24.295485 7231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:24.323364 7231 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.195:44947
I20260924 12:46:24.323448 7357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.195:44947 every 8 connection(s)
I20260924 12:46:24.324673 7231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260924 12:46:24.328817 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 7231
I20260924 12:46:24.329031 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260924 12:46:24.338591 7358 heartbeater.cc:347] Connected to a master server at 127.5.77.254:46369
I20260924 12:46:24.338888 7358 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:24.339579 7358 heartbeater.cc:511] Master 127.5.77.254:46369 requested a full tablet report, sending...
I20260924 12:46:24.340936 6911 ts_manager.cc:194] Registered new tserver with Master: 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:24.341671 6911 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.195:54545
I20260924 12:46:24.344503 5431 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:46:24.347568 5431 test_util.cc:268] Using random seed: -53774970
I20260924 12:46:24.365613 6911 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33400:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:46:24.367172 6911 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:46:24.392570 7163 tablet_service.cc:1467] Processing CreateTablet for tablet 6423621b78684850a4cb8a6a75a63996 (DEFAULT_TABLE table=test-workload [id=4d86b1005d9f47f480460059eee4cbc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:24.393822 7163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6423621b78684850a4cb8a6a75a63996. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:24.393476 7293 tablet_service.cc:1467] Processing CreateTablet for tablet 6423621b78684850a4cb8a6a75a63996 (DEFAULT_TABLE table=test-workload [id=4d86b1005d9f47f480460059eee4cbc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:24.393579 7032 tablet_service.cc:1467] Processing CreateTablet for tablet 6423621b78684850a4cb8a6a75a63996 (DEFAULT_TABLE table=test-workload [id=4d86b1005d9f47f480460059eee4cbc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:24.394831 7032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6423621b78684850a4cb8a6a75a63996. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:24.395562 7293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6423621b78684850a4cb8a6a75a63996. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:24.405041 7376 tablet_bootstrap.cc:492] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d: Bootstrap starting.
I20260924 12:46:24.408190 7377 tablet_bootstrap.cc:492] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde: Bootstrap starting.
I20260924 12:46:24.409596 7378 tablet_bootstrap.cc:492] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e: Bootstrap starting.
I20260924 12:46:24.410257 7376 tablet_bootstrap.cc:654] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:24.411685 7376 log.cc:824] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:24.411857 7377 tablet_bootstrap.cc:654] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:24.412369 7378 tablet_bootstrap.cc:654] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:24.413080 7377 log.cc:824] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:24.413354 7378 log.cc:824] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:24.414857 7376 tablet_bootstrap.cc:492] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d: No bootstrap required, opened a new log
I20260924 12:46:24.415217 7376 ts_tablet_manager.cc:1397] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260924 12:46:24.415509 7377 tablet_bootstrap.cc:492] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde: No bootstrap required, opened a new log
I20260924 12:46:24.415755 7377 ts_tablet_manager.cc:1397] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 12:46:24.416157 7378 tablet_bootstrap.cc:492] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e: No bootstrap required, opened a new log
I20260924 12:46:24.416450 7378 ts_tablet_manager.cc:1397] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 12:46:24.423103 7377 raft_consensus.cc:348] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:24.423518 7377 raft_consensus.cc:374] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:24.423610 7377 raft_consensus.cc:729] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db3b3e723eb41299037b5eab1b00fde, State: Initialized, Role: FOLLOWER
I20260924 12:46:24.424247 7377 consensus_queue.cc:260] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:24.426230 7377 ts_tablet_manager.cc:1428] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260924 12:46:24.426541 7376 raft_consensus.cc:348] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:24.427098 7376 raft_consensus.cc:374] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:24.427217 7376 raft_consensus.cc:729] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fb4103951824ad892bfc6e460c4069d, State: Initialized, Role: FOLLOWER
I20260924 12:46:24.427184 7378 raft_consensus.cc:348] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:24.427630 7378 raft_consensus.cc:374] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:24.427735 7378 raft_consensus.cc:729] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7632672814be4a688ec7ab18a503a58e, State: Initialized, Role: FOLLOWER
I20260924 12:46:24.428691 7097 heartbeater.cc:503] Master 127.5.77.254:46369 was elected leader, sending a full tablet report...
I20260924 12:46:24.430275 7383 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:46:24.430718 7383 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:24.436131 7383 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:24.438575 7378 consensus_queue.cc:260] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:24.439301 7385 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:46:24.439597 7385 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
W20260924 12:46:24.440286 6987 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.5.77.194:35473 for RPC RequestConsensusVote timed out after 0.007s (ON_OUTBOUND_QUEUE)
I20260924 12:46:24.447446 7358 heartbeater.cc:503] Master 127.5.77.254:46369 was elected leader, sending a full tablet report...
I20260924 12:46:24.447784 7183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:24.448714 7385 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435)
I20260924 12:46:24.449596 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:24.450008 7313 raft_consensus.cc:2454] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9db3b3e723eb41299037b5eab1b00fde in term 0.
I20260924 12:46:24.449880 7376 consensus_queue.cc:260] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:24.450655 7183 raft_consensus.cc:2454] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9db3b3e723eb41299037b5eab1b00fde in term 0.
I20260924 12:46:24.452311 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde; no voters:
I20260924 12:46:24.453054 7384 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:46:24.453298 7384 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
W20260924 12:46:24.461030 7248 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.5.77.194:35473 timed out after 0.006s (ON_OUTBOUND_QUEUE)
I20260924 12:46:24.467083 7383 raft_consensus.cc:2781] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:46:24.467376 7383 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:46:24.467545 7383 raft_consensus.cc:3037] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:24.467945 7378 ts_tablet_manager.cc:1428] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e: Time spent starting tablet: real 0.051s user 0.016s sys 0.001s
I20260924 12:46:24.468730 7183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "7632672814be4a688ec7ab18a503a58e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:24.468979 7183 raft_consensus.cc:2454] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7632672814be4a688ec7ab18a503a58e in term 0.
I20260924 12:46:24.469470 7384 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:24.470202 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "7632672814be4a688ec7ab18a503a58e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:24.471263 7228 heartbeater.cc:503] Master 127.5.77.254:46369 was elected leader, sending a full tablet report...
I20260924 12:46:24.473984 7383 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
W20260924 12:46:24.474882 7098 tablet.cc:2369] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:46:24.479099 7383 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 1 election: Requested vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:24.479646 7052 raft_consensus.cc:2379] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7632672814be4a688ec7ab18a503a58e in current term 1: Already voted for candidate 9db3b3e723eb41299037b5eab1b00fde in this term.
W20260924 12:46:24.480480 7052 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.5.77.195:57505 (request call id 1) took 10 ms (client timeout 6 ms). Trace:
W20260924 12:46:24.480635 7052 rpcz_store.cc:269] 0924 12:46:24.469673 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:24.469736 (+ 63us) service_pool.cc:227] Handling call
0924 12:46:24.480223 (+ 10487us) spinlock_profiling.cc:243] Waited 8.75 ms on lock 0x615000072c24. stack: 00007f9ece5e966e 00007f9ece5e9430 00007f9ecd6705ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007f9ed9b15d0a 00007f9edd6af5eb 00007f9ed7f99e06 00007f9ed7f99b35 00007f9ed51259be 00007f9ed512479e 00007f9ed512ab25 00007f9ed512b54f 00007f9ed512b378 00000000009edb7c
0924 12:46:24.480406 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20133248}
I20260924 12:46:24.483585 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e"
I20260924 12:46:24.483810 7313 raft_consensus.cc:3037] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:24.483989 7376 ts_tablet_manager.cc:1428] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d: Time spent starting tablet: real 0.069s user 0.040s sys 0.027s
I20260924 12:46:24.485455 7183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d"
I20260924 12:46:24.485692 7183 raft_consensus.cc:3037] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:24.486825 7313 raft_consensus.cc:2454] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9db3b3e723eb41299037b5eab1b00fde in term 1.
I20260924 12:46:24.487336 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde; no voters:
I20260924 12:46:24.487649 7383 raft_consensus.cc:2781] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:24.487805 7383 raft_consensus.cc:686] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 1 LEADER]: Becoming Leader. State: Replica: 9db3b3e723eb41299037b5eab1b00fde, State: Running, Role: LEADER
I20260924 12:46:24.488339 7383 consensus_queue.cc:237] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:24.489553 7183 raft_consensus.cc:2454] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9db3b3e723eb41299037b5eab1b00fde in term 1.
W20260924 12:46:24.482775 7248 leader_election.cc:334] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473): Timed out: RequestConsensusVote RPC to 127.5.77.194:35473 timed out after 0.006s (SENT)
W20260924 12:46:24.490262 7248 leader_election.cc:334] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435): Timed out: RequestConsensusVote RPC to 127.5.77.193:36435 timed out after 0.006s (SENT)
I20260924 12:46:24.490386 7248 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7632672814be4a688ec7ab18a503a58e; no voters: 6fb4103951824ad892bfc6e460c4069d, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:24.490880 7388 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 12:46:24.492439 7118 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.5.77.193:36435 for RPC RequestConsensusVote timed out after 0.007s (ON_OUTBOUND_QUEUE)
I20260924 12:46:24.493472 7183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d"
I20260924 12:46:24.493721 7183 raft_consensus.cc:2362] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 1 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9db3b3e723eb41299037b5eab1b00fde in term 1. Re-sending same reply.
I20260924 12:46:24.495530 6911 catalog_manager.cc:5495] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193). New cstate: current_term: 1 leader_uuid: "9db3b3e723eb41299037b5eab1b00fde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } } }
I20260924 12:46:24.500413 7388 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9db3b3e723eb41299037b5eab1b00fde)
I20260924 12:46:24.500794 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:24.500658 7388 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:24.501390 7313 raft_consensus.cc:2379] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6fb4103951824ad892bfc6e460c4069d in current term 1: Already voted for candidate 9db3b3e723eb41299037b5eab1b00fde in this term.
I20260924 12:46:24.501456 7388 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435)
I20260924 12:46:24.502157 7182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "7632672814be4a688ec7ab18a503a58e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:24.502437 7182 raft_consensus.cc:2454] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7632672814be4a688ec7ab18a503a58e in term 1.
I20260924 12:46:24.502952 7248 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e; no voters:
I20260924 12:46:24.503286 7051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "7632672814be4a688ec7ab18a503a58e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:24.503437 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:24.504330 7388 raft_consensus.cc:2781] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 12:46:24.504653 7388 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9db3b3e723eb41299037b5eab1b00fde)
I20260924 12:46:24.504791 7388 raft_consensus.cc:3037] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:46:24.504777 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:24.505144 7391 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 12:46:24.505420 7382 consensus_queue.cc:1035] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:24.507723 7388 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:24.508757 7182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "7632672814be4a688ec7ab18a503a58e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d"
I20260924 12:46:24.509488 7313 raft_consensus.cc:1226] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [term 2 FOLLOWER]: Rejecting Update request from peer 9db3b3e723eb41299037b5eab1b00fde for earlier term 1. Current term is 2. Ops: []
I20260924 12:46:24.510998 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "7632672814be4a688ec7ab18a503a58e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde"
I20260924 12:46:24.511602 7248 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7632672814be4a688ec7ab18a503a58e; no voters: 6fb4103951824ad892bfc6e460c4069d, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:24.511886 7385 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 12:46:24.512077 7388 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [CANDIDATE]: Term 2 election: Requested vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435)
I20260924 12:46:24.512537 7382 consensus_queue.cc:1046] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:24.514897 7313 raft_consensus.cc:1226] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [term 2 FOLLOWER]: Rejecting Update request from peer 9db3b3e723eb41299037b5eab1b00fde for earlier term 1. Current term is 2. Ops: []
I20260924 12:46:24.515203 7391 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9db3b3e723eb41299037b5eab1b00fde)
I20260924 12:46:24.515406 7391 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:24.516420 7391 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:24.516858 7401 raft_consensus.cc:3032] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 1 LEADER]: Stepping down as leader of term 1
I20260924 12:46:24.516979 7401 raft_consensus.cc:729] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9db3b3e723eb41299037b5eab1b00fde, State: Running, Role: LEADER
I20260924 12:46:24.517050 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:24.517146 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:24.517375 7313 raft_consensus.cc:2379] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6fb4103951824ad892bfc6e460c4069d in current term 2: Already voted for candidate 7632672814be4a688ec7ab18a503a58e in this term.
I20260924 12:46:24.517311 7401 consensus_queue.cc:260] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:24.517918 7401 raft_consensus.cc:3037] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 1 FOLLOWER]: Advancing to term 2
W20260924 12:46:24.517827 7382 consensus_queue.cc:1159] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6fb4103951824ad892bfc6e460c4069d" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20260924 12:46:24.518582 7388 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:46:24.518718 7388 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:24.519215 7388 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435)
I20260924 12:46:24.519711 7183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "7632672814be4a688ec7ab18a503a58e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:24.519775 7051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "7632672814be4a688ec7ab18a503a58e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:24.519912 7183 raft_consensus.cc:2454] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7632672814be4a688ec7ab18a503a58e in term 1.
I20260924 12:46:24.520268 7248 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e; no voters:
I20260924 12:46:24.520694 7388 raft_consensus.cc:2781] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 12:46:24.520779 7388 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:46:24.520817 7052 raft_consensus.cc:2454] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fb4103951824ad892bfc6e460c4069d in term 2.
I20260924 12:46:24.520887 7388 raft_consensus.cc:3037] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [term 2 FOLLOWER]: Advancing to term 3
I20260924 12:46:24.521328 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d, 9db3b3e723eb41299037b5eab1b00fde; no voters: 7632672814be4a688ec7ab18a503a58e
I20260924 12:46:24.521744 7391 raft_consensus.cc:2781] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 12:46:24.521834 7391 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9db3b3e723eb41299037b5eab1b00fde)
I20260924 12:46:24.521934 7391 raft_consensus.cc:3037] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:46:24.523473 7388 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:24.524339 7183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "7632672814be4a688ec7ab18a503a58e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d"
I20260924 12:46:24.524629 7391 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:24.525075 7183 raft_consensus.cc:3037] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 2 FOLLOWER]: Advancing to term 3
I20260924 12:46:24.525576 7391 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 2 election: Requested vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:24.525841 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e"
I20260924 12:46:24.526086 7313 raft_consensus.cc:2354] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6fb4103951824ad892bfc6e460c4069d for earlier term 2. Current term is 3.
I20260924 12:46:24.526408 7051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde"
I20260924 12:46:24.526651 7118 leader_election.cc:398] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 2 election: Vote denied by peer 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947) with higher term. Message: Invalid argument: T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6fb4103951824ad892bfc6e460c4069d for earlier term 2. Current term is 3.
I20260924 12:46:24.526760 7118 leader_election.cc:401] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260924 12:46:24.527575 7183 raft_consensus.cc:2454] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7632672814be4a688ec7ab18a503a58e in term 3.
I20260924 12:46:24.527812 7391 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947) with higher term. Message: Invalid argument: T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6fb4103951824ad892bfc6e460c4069d for earlier term 2. Current term is 3.
I20260924 12:46:24.528053 7248 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e; no voters:
I20260924 12:46:24.528151 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "7632672814be4a688ec7ab18a503a58e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde"
I20260924 12:46:24.528523 7385 raft_consensus.cc:2781] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [term 3 FOLLOWER]: Leader election won for term 3
I20260924 12:46:24.529114 7385 raft_consensus.cc:686] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [term 3 LEADER]: Becoming Leader. State: Replica: 7632672814be4a688ec7ab18a503a58e, State: Running, Role: LEADER
I20260924 12:46:24.529266 7051 raft_consensus.cc:2454] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fb4103951824ad892bfc6e460c4069d in term 2.
I20260924 12:46:24.529517 7388 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [CANDIDATE]: Term 3 election: Requested vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435)
I20260924 12:46:24.529613 7385 consensus_queue.cc:237] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:24.532891 7051 raft_consensus.cc:3037] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 2 FOLLOWER]: Advancing to term 3
I20260924 12:46:24.532776 7183 raft_consensus.cc:1261] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Refusing update from remote peer 7632672814be4a688ec7ab18a503a58e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 12:46:24.533401 7388 consensus_queue.cc:1035] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:24.533452 6911 catalog_manager.cc:5495] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e reported cstate change: term changed from 1 to 3, leader changed from 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193) to 7632672814be4a688ec7ab18a503a58e (127.5.77.195). New cstate: current_term: 3 leader_uuid: "7632672814be4a688ec7ab18a503a58e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } health_report { overall_health: HEALTHY } } }
I20260924 12:46:24.535631 7051 raft_consensus.cc:1261] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Refusing update from remote peer 7632672814be4a688ec7ab18a503a58e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 12:46:24.536612 7385 consensus_queue.cc:1035] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 12:46:24.537253 7401 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:46:24.537402 7401 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:24.538040 7401 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:24.538834 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:24.540024 7182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:24.540512 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9db3b3e723eb41299037b5eab1b00fde; no voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e
I20260924 12:46:24.540859 7382 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 12:46:24.559338 7012 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41870: Illegal state: replica 9db3b3e723eb41299037b5eab1b00fde is not leader of this config: current role FOLLOWER
W20260924 12:46:24.559312 7011 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41870: Illegal state: replica 9db3b3e723eb41299037b5eab1b00fde is not leader of this config: current role FOLLOWER
W20260924 12:46:24.574493 7143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46560: Illegal state: replica 6fb4103951824ad892bfc6e460c4069d is not leader of this config: current role FOLLOWER
W20260924 12:46:24.574512 7142 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46560: Illegal state: replica 6fb4103951824ad892bfc6e460c4069d is not leader of this config: current role FOLLOWER
W20260924 12:46:24.592780 7411 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260924 12:46:24.592912 7411 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20260924 12:46:24.719111 7183 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260924 12:46:24.719127 7051 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260924 12:46:24.719208 7051 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20260924 12:46:24.719216 7183 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260924 12:46:24.833330 7391 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:24.833540 7391 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:24.834424 7391 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:24.835062 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:24.835103 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:24.835808 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:24.836195 7391 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:24.884117 7391 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:24.884307 7391 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:24.884992 7391 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:24.885601 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:24.886685 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:24.887213 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:24.887462 7391 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:24.894663 7391 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:24.894862 7391 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:24.895532 7391 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:24.896054 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:24.896072 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:24.896800 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:24.897189 7391 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:25.367488 7401 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:25.367718 7401 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:25.368501 7401 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:25.368968 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:25.369582 7183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:25.370523 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9db3b3e723eb41299037b5eab1b00fde; no voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e
I20260924 12:46:25.370798 7401 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:25.650748 7401 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:25.650981 7401 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:25.651682 7401 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:25.652244 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:25.652240 7182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:25.653261 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9db3b3e723eb41299037b5eab1b00fde; no voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e
I20260924 12:46:25.653584 7401 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:25.654047 7422 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48094 (ReqId={client: b34b87926d454675a830c7cf36b10cff, seq_no=1, attempt_no=2}) took 1067 ms. Trace:
I20260924 12:46:25.654203 7422 rpcz_store.cc:276] 0924 12:46:24.586730 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:24.586941 (+ 211us) service_pool.cc:227] Handling call
0924 12:46:25.653990 (+1067049us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:46:24.718027 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:24.718155 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:24.718162 (+ 7us) write_op.cc:435] Acquired schema lock
0924 12:46:24.718165 (+ 3us) tablet.cc:660] Decoding operations
0924 12:46:24.718261 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:24.718294 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:24.718298 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:24.718355 (+ 57us) tablet.cc:699] Row locks acquired
0924 12:46:24.718357 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:46:24.718410 (+ 53us) write_op.cc:270] Start()
0924 12:46:25.531571 (+813161us) write_op.cc:276] Timestamp: P: 1790253984718350 usec, L: 0
0924 12:46:25.531575 (+ 4us) op_driver.cc:331] REPLICATION: starting
0924 12:46:25.531841 (+ 266us) log.cc:842] Serialized 345 byte log entry
0924 12:46:25.651883 (+120042us) op_driver.cc:444] REPLICATION: finished
0924 12:46:25.653427 (+ 1544us) write_op.cc:301] APPLY: starting
0924 12:46:25.653444 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0924 12:46:25.653478 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0924 12:46:25.653507 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:46:25.653606 (+ 99us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:46:25.653638 (+ 32us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:46:25.653645 (+ 7us) write_op.cc:312] APPLY: finished
0924 12:46:25.653749 (+ 104us) log.cc:842] Serialized 22 byte log entry
0924 12:46:25.653860 (+ 111us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:46:25.653876 (+ 16us) write_op.cc:454] Released schema lock
0924 12:46:25.653883 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":896,"child_traces":[["op",{"apply.queue_time_us":1475,"mutex_wait_us":37,"num_ops":1,"prepare.queue_time_us":127487,"prepare.run_cpu_time_us":940,"prepare.run_wall_time_us":814044,"raft.queue_time_us":226,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":4,"replication_time_us":120272,"spinlock_wait_cycles":261376,"thread_start_us":1353,"threads_started":1}]]}
W20260924 12:46:25.656777 7407 scanner-internal.cc:458] Time spent opening tablet: real 0.663s user 0.001s sys 0.000s
W20260924 12:46:25.656801 7408 scanner-internal.cc:458] Time spent opening tablet: real 0.662s user 0.001s sys 0.000s
I20260924 12:46:26.299206 7421 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:26.299459 7421 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:26.300246 7421 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:26.300755 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:26.300802 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:26.301704 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:26.722370 7423 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:26.722646 7423 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:26.723308 7423 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:26.723855 7182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:26.723984 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:26.724658 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9db3b3e723eb41299037b5eab1b00fde; no voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e
I20260924 12:46:26.726490 7430 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48094 (ReqId={client: b34b87926d454675a830c7cf36b10cff, seq_no=2, attempt_no=0}) took 1885 ms. Trace:
I20260924 12:46:26.726650 7430 rpcz_store.cc:276] 0924 12:46:24.840798 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:24.840852 (+ 54us) service_pool.cc:227] Handling call
0924 12:46:26.726392 (+1885540us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:46:25.532097 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:25.532224 (+ 127us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:25.532230 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:46:25.532232 (+ 2us) tablet.cc:660] Decoding operations
0924 12:46:25.532331 (+ 99us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:25.532342 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:25.532347 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:25.532403 (+ 56us) tablet.cc:699] Row locks acquired
0924 12:46:25.532405 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:46:25.532452 (+ 47us) write_op.cc:270] Start()
0924 12:46:26.154590 (+622138us) write_op.cc:276] Timestamp: P: 1790253985532410 usec, L: 0
0924 12:46:26.154594 (+ 4us) op_driver.cc:331] REPLICATION: starting
0924 12:46:26.154856 (+ 262us) log.cc:842] Serialized 354 byte log entry
0924 12:46:26.723842 (+568986us) op_driver.cc:444] REPLICATION: finished
0924 12:46:26.724613 (+ 771us) write_op.cc:301] APPLY: starting
0924 12:46:26.724633 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0924 12:46:26.724678 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0924 12:46:26.724708 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:46:26.724782 (+ 74us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:46:26.724816 (+ 34us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:46:26.724822 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:46:26.724945 (+ 123us) log.cc:842] Serialized 22 byte log entry
0924 12:46:26.726253 (+ 1308us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:46:26.726278 (+ 25us) write_op.cc:454] Released schema lock
0924 12:46:26.726284 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":698,"num_ops":1,"prepare.queue_time_us":690371,"prepare.run_cpu_time_us":781,"prepare.run_wall_time_us":622835,"replication_time_us":569221,"thread_start_us":1476,"threads_started":1}]]}
W20260924 12:46:26.727384 7408 scanner-internal.cc:458] Time spent opening tablet: real 0.920s user 0.001s sys 0.000s
W20260924 12:46:26.727509 7407 scanner-internal.cc:458] Time spent opening tablet: real 0.920s user 0.001s sys 0.000s
I20260924 12:46:27.189073 7423 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:27.190711 7430 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48094 (ReqId={client: b34b87926d454675a830c7cf36b10cff, seq_no=3, attempt_no=0}) took 1533 ms. Trace:
I20260924 12:46:27.190845 7430 rpcz_store.cc:276] 0924 12:46:25.657210 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:25.657275 (+ 65us) service_pool.cc:227] Handling call
0924 12:46:27.190641 (+1533366us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:46:26.154972 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:26.155076 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:26.155080 (+ 4us) write_op.cc:435] Acquired schema lock
0924 12:46:26.155082 (+ 2us) tablet.cc:660] Decoding operations
0924 12:46:26.155162 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:26.155174 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:26.155198 (+ 24us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:26.155254 (+ 56us) tablet.cc:699] Row locks acquired
0924 12:46:26.155256 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:46:26.155295 (+ 39us) write_op.cc:270] Start()
0924 12:46:26.562519 (+407224us) write_op.cc:276] Timestamp: P: 1790253986155246 usec, L: 0
0924 12:46:26.562527 (+ 8us) op_driver.cc:331] REPLICATION: starting
0924 12:46:26.562785 (+ 258us) log.cc:842] Serialized 354 byte log entry
0924 12:46:27.189980 (+627195us) op_driver.cc:444] REPLICATION: finished
0924 12:46:27.190098 (+ 118us) write_op.cc:301] APPLY: starting
0924 12:46:27.190118 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0924 12:46:27.190161 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0924 12:46:27.190209 (+ 48us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:46:27.190282 (+ 73us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:46:27.190315 (+ 33us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:46:27.190321 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:46:27.190417 (+ 96us) log.cc:842] Serialized 22 byte log entry
0924 12:46:27.190530 (+ 113us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:46:27.190548 (+ 18us) write_op.cc:454] Released schema lock
0924 12:46:27.190554 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":49,"num_ops":1,"prepare.queue_time_us":497299,"prepare.run_cpu_time_us":806,"prepare.run_wall_time_us":407893,"replication_time_us":627423}]]}
I20260924 12:46:27.228938 7421 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:27.924808 7421 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:27.924994 7421 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:27.925742 7421 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:27.926172 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:27.926234 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:27.926831 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:27.966503 7421 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:27.968904 7437 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48094 (ReqId={client: b34b87926d454675a830c7cf36b10cff, seq_no=4, attempt_no=0}) took 1240 ms. Trace:
I20260924 12:46:27.969098 7437 rpcz_store.cc:276] 0924 12:46:26.728338 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:26.728395 (+ 57us) service_pool.cc:227] Handling call
0924 12:46:27.968830 (+1240435us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:46:26.728845 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:26.728929 (+ 84us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:26.728933 (+ 4us) write_op.cc:435] Acquired schema lock
0924 12:46:26.728935 (+ 2us) tablet.cc:660] Decoding operations
0924 12:46:26.729003 (+ 68us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:26.729011 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:26.729014 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:26.729063 (+ 49us) tablet.cc:699] Row locks acquired
0924 12:46:26.729065 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:46:26.729094 (+ 29us) write_op.cc:270] Start()
0924 12:46:27.492214 (+763120us) write_op.cc:276] Timestamp: P: 1790253986729047 usec, L: 0
0924 12:46:27.492218 (+ 4us) op_driver.cc:331] REPLICATION: starting
0924 12:46:27.492473 (+ 255us) log.cc:842] Serialized 354 byte log entry
0924 12:46:27.967743 (+475270us) op_driver.cc:444] REPLICATION: finished
0924 12:46:27.968272 (+ 529us) write_op.cc:301] APPLY: starting
0924 12:46:27.968289 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0924 12:46:27.968325 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0924 12:46:27.968351 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:46:27.968443 (+ 92us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:46:27.968503 (+ 60us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:46:27.968510 (+ 7us) write_op.cc:312] APPLY: finished
0924 12:46:27.968605 (+ 95us) log.cc:842] Serialized 22 byte log entry
0924 12:46:27.968704 (+ 99us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:46:27.968723 (+ 19us) write_op.cc:454] Released schema lock
0924 12:46:27.968730 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":451,"num_ops":1,"prepare.queue_time_us":64,"prepare.run_cpu_time_us":645,"prepare.run_wall_time_us":763698,"replication_time_us":475499,"thread_start_us":265,"threads_started":1}]]}
I20260924 12:46:27.970521 7273 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:48094 (request call id 11) took 1091 ms. Trace:
I20260924 12:46:27.970592 7272 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:48094 (request call id 12) took 1091 ms. Trace:
I20260924 12:46:27.970700 7272 rpcz_store.cc:276] 0924 12:46:26.879043 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:26.879101 (+ 58us) service_pool.cc:227] Handling call
0924 12:46:26.879250 (+ 149us) tablet_service.cc:2780] Created scanner 8c91497accb542739d228e01c9f0d21a for tablet 6423621b78684850a4cb8a6a75a63996, query id is e6786675db60459cb296cbae2e85c783
0924 12:46:26.880198 (+ 948us) tablet_service.cc:2920] Creating iterator
0924 12:46:26.880236 (+ 38us) tablet_service.cc:3297] Waiting safe time to advance
0924 12:46:27.492361 (+612125us) tablet_service.cc:3304] Waiting for operations to commit
0924 12:46:27.968805 (+476444us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1088557 microseconds
0924 12:46:27.968873 (+ 68us) tablet_service.cc:2945] Iterator created
0924 12:46:27.969510 (+ 637us) tablet_service.cc:2967] Iterator init: OK
0924 12:46:27.969548 (+ 38us) tablet_service.cc:3010] has_more: true
0924 12:46:27.969582 (+ 34us) tablet_service.cc:3027] Continuing scan request
0924 12:46:27.969674 (+ 92us) tablet_service.cc:3091] Found scanner 8c91497accb542739d228e01c9f0d21a for tablet 6423621b78684850a4cb8a6a75a63996, query id is e6786675db60459cb296cbae2e85c783
0924 12:46:27.970527 (+ 853us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":59264}
I20260924 12:46:27.970650 7273 rpcz_store.cc:276] 0924 12:46:26.878890 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:26.878984 (+ 94us) service_pool.cc:227] Handling call
0924 12:46:26.879249 (+ 265us) tablet_service.cc:2780] Created scanner 2a5d79eaaacc4892b3089b864be98a6e for tablet 6423621b78684850a4cb8a6a75a63996, query id is 4e4f136e4fa44bf3a08bc28b261c94bc
0924 12:46:26.880187 (+ 938us) tablet_service.cc:2920] Creating iterator
0924 12:46:26.880236 (+ 49us) tablet_service.cc:3297] Waiting safe time to advance
0924 12:46:27.492350 (+612114us) tablet_service.cc:3304] Waiting for operations to commit
0924 12:46:27.968742 (+476392us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1088488 microseconds
0924 12:46:27.968873 (+ 131us) tablet_service.cc:2945] Iterator created
0924 12:46:27.969464 (+ 591us) tablet_service.cc:2967] Iterator init: OK
0924 12:46:27.969501 (+ 37us) tablet_service.cc:3010] has_more: true
0924 12:46:27.969538 (+ 37us) tablet_service.cc:3027] Continuing scan request
0924 12:46:27.969614 (+ 76us) tablet_service.cc:3091] Found scanner 2a5d79eaaacc4892b3089b864be98a6e for tablet 6423621b78684850a4cb8a6a75a63996, query id is 4e4f136e4fa44bf3a08bc28b261c94bc
0924 12:46:27.970431 (+ 817us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":16384}
W20260924 12:46:27.971462 7407 scanner-internal.cc:458] Time spent opening tablet: real 1.093s user 0.001s sys 0.000s
W20260924 12:46:27.971779 7408 scanner-internal.cc:458] Time spent opening tablet: real 1.094s user 0.001s sys 0.000s
I20260924 12:46:28.381961 7421 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:28.382154 7421 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:28.382975 7421 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:28.383392 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:28.383751 7437 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48094 (ReqId={client: b34b87926d454675a830c7cf36b10cff, seq_no=5, attempt_no=0}) took 1191 ms. Trace:
I20260924 12:46:28.383881 7437 rpcz_store.cc:276] 0924 12:46:27.192560 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:27.192621 (+ 61us) service_pool.cc:227] Handling call
0924 12:46:28.383692 (+1191071us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:46:27.492580 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:27.492698 (+ 118us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:27.492704 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:46:27.492706 (+ 2us) tablet.cc:660] Decoding operations
0924 12:46:27.492801 (+ 95us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:27.492813 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:27.492818 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:27.492872 (+ 54us) tablet.cc:699] Row locks acquired
0924 12:46:27.492875 (+ 3us) write_op.cc:260] PREPARE: finished
0924 12:46:27.492913 (+ 38us) write_op.cc:270] Start()
0924 12:46:28.042033 (+549120us) write_op.cc:276] Timestamp: P: 1790253987492880 usec, L: 0
0924 12:46:28.042038 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:46:28.042213 (+ 175us) log.cc:842] Serialized 354 byte log entry
0924 12:46:28.383079 (+340866us) op_driver.cc:444] REPLICATION: finished
0924 12:46:28.383193 (+ 114us) write_op.cc:301] APPLY: starting
0924 12:46:28.383213 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0924 12:46:28.383256 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0924 12:46:28.383280 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:46:28.383351 (+ 71us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:46:28.383387 (+ 36us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:46:28.383393 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:46:28.383485 (+ 92us) log.cc:842] Serialized 22 byte log entry
0924 12:46:28.383575 (+ 90us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:46:28.383595 (+ 20us) write_op.cc:454] Released schema lock
0924 12:46:28.383601 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":58,"num_ops":1,"prepare.queue_time_us":299486,"prepare.run_cpu_time_us":688,"prepare.run_wall_time_us":549743,"raft.queue_time_us":49,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":145,"replication_time_us":341012,"spinlock_wait_cycles":258944}]]}
I20260924 12:46:28.384366 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:28.384883 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:28.385146 7421 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:28.422806 7423 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:28.423002 7423 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:28.423717 7423 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:28.424124 7182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:28.424173 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:28.424777 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9db3b3e723eb41299037b5eab1b00fde; no voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e
I20260924 12:46:28.425164 7423 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:28.827522 7423 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:28.827729 7423 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:28.828552 7423 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:28.829149 7183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:28.829331 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:28.829933 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9db3b3e723eb41299037b5eab1b00fde; no voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e
I20260924 12:46:29.086189 7421 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:29.086412 7421 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:29.087198 7421 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:29.087832 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 9 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:29.087848 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 9 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:29.088598 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:29.089466 7443 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48094 (ReqId={client: b34b87926d454675a830c7cf36b10cff, seq_no=6, attempt_no=0}) took 1117 ms. Trace:
I20260924 12:46:29.089704 7443 rpcz_store.cc:276] 0924 12:46:27.971683 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:27.971743 (+ 60us) service_pool.cc:227] Handling call
0924 12:46:29.089392 (+1117649us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:46:28.042360 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:28.042516 (+ 156us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:28.042522 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:46:28.042525 (+ 3us) tablet.cc:660] Decoding operations
0924 12:46:28.042623 (+ 98us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:28.042635 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:28.042640 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:28.042698 (+ 58us) tablet.cc:699] Row locks acquired
0924 12:46:28.042701 (+ 3us) write_op.cc:260] PREPARE: finished
0924 12:46:28.042753 (+ 52us) write_op.cc:270] Start()
0924 12:46:28.579896 (+537143us) write_op.cc:276] Timestamp: P: 1790253988042696 usec, L: 0
0924 12:46:28.579900 (+ 4us) op_driver.cc:331] REPLICATION: starting
0924 12:46:28.580188 (+ 288us) log.cc:842] Serialized 354 byte log entry
0924 12:46:29.087585 (+507397us) op_driver.cc:444] REPLICATION: finished
0924 12:46:29.088783 (+ 1198us) write_op.cc:301] APPLY: starting
0924 12:46:29.088806 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0924 12:46:29.088850 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0924 12:46:29.088898 (+ 48us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:46:29.089015 (+ 117us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:46:29.089047 (+ 32us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:46:29.089052 (+ 5us) write_op.cc:312] APPLY: finished
0924 12:46:29.089134 (+ 82us) log.cc:842] Serialized 22 byte log entry
0924 12:46:29.089259 (+ 125us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:46:29.089281 (+ 22us) write_op.cc:454] Released schema lock
0924 12:46:29.089289 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1119,"num_ops":1,"prepare.queue_time_us":70195,"prepare.run_cpu_time_us":830,"prepare.run_wall_time_us":537900,"replication_time_us":507654,"thread_start_us":1033,"threads_started":1}]]}
W20260924 12:46:29.091403 7408 scanner-internal.cc:458] Time spent opening tablet: real 0.969s user 0.000s sys 0.000s
W20260924 12:46:29.092206 7407 scanner-internal.cc:458] Time spent opening tablet: real 0.970s user 0.001s sys 0.000s
I20260924 12:46:29.283394 7423 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:29.752146 7421 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:29.755101 7448 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48094 (ReqId={client: b34b87926d454675a830c7cf36b10cff, seq_no=7, attempt_no=0}) took 1369 ms. Trace:
I20260924 12:46:29.755261 7448 rpcz_store.cc:276] 0924 12:46:28.385888 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:28.385939 (+ 51us) service_pool.cc:227] Handling call
0924 12:46:29.755028 (+1369089us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:46:28.580305 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:28.580422 (+ 117us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:28.580429 (+ 7us) write_op.cc:435] Acquired schema lock
0924 12:46:28.580431 (+ 2us) tablet.cc:660] Decoding operations
0924 12:46:28.580539 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:28.580553 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:28.580558 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:28.580614 (+ 56us) tablet.cc:699] Row locks acquired
0924 12:46:28.580616 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:46:28.580661 (+ 45us) write_op.cc:270] Start()
0924 12:46:29.073814 (+493153us) write_op.cc:276] Timestamp: P: 1790253988580627 usec, L: 0
0924 12:46:29.073821 (+ 7us) op_driver.cc:331] REPLICATION: starting
0924 12:46:29.074100 (+ 279us) log.cc:842] Serialized 354 byte log entry
0924 12:46:29.753405 (+679305us) op_driver.cc:444] REPLICATION: finished
0924 12:46:29.754454 (+ 1049us) write_op.cc:301] APPLY: starting
0924 12:46:29.754502 (+ 48us) tablet.cc:1365] starting BulkCheckPresence
0924 12:46:29.754549 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0924 12:46:29.754584 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:46:29.754687 (+ 103us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:46:29.754725 (+ 38us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:46:29.754731 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:46:29.754817 (+ 86us) log.cc:842] Serialized 22 byte log entry
0924 12:46:29.754904 (+ 87us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:46:29.754924 (+ 20us) write_op.cc:454] Released schema lock
0924 12:46:29.754930 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":966,"num_ops":1,"prepare.queue_time_us":193943,"prepare.run_cpu_time_us":809,"prepare.run_wall_time_us":493870,"replication_time_us":679556,"thread_start_us":904,"threads_started":1}]]}
I20260924 12:46:29.895423 7423 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:29.895623 7423 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:29.896358 7423 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:29.896962 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:29.896996 7183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:29.897696 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9db3b3e723eb41299037b5eab1b00fde; no voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e
I20260924 12:46:30.272745 7421 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:30.272964 7421 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:30.273733 7421 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:30.274261 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 11 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:30.274704 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 11 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:30.275296 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:30.275995 7421 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:30.276516 7454 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48094 (ReqId={client: b34b87926d454675a830c7cf36b10cff, seq_no=8, attempt_no=0}) took 1184 ms. Trace:
I20260924 12:46:30.276631 7454 rpcz_store.cc:276] 0924 12:46:29.091723 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:29.091803 (+ 80us) service_pool.cc:227] Handling call
0924 12:46:30.276451 (+1184648us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:46:29.092256 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:29.092352 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:29.092357 (+ 5us) write_op.cc:435] Acquired schema lock
0924 12:46:29.092359 (+ 2us) tablet.cc:660] Decoding operations
0924 12:46:29.092441 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:29.092451 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:29.092455 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:29.092502 (+ 47us) tablet.cc:699] Row locks acquired
0924 12:46:29.092504 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:46:29.092536 (+ 32us) write_op.cc:270] Start()
0924 12:46:29.739686 (+647150us) write_op.cc:276] Timestamp: P: 1790253989092506 usec, L: 0
0924 12:46:29.739691 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:46:29.740078 (+ 387us) log.cc:842] Serialized 354 byte log entry
0924 12:46:30.274318 (+534240us) op_driver.cc:444] REPLICATION: finished
0924 12:46:30.274986 (+ 668us) write_op.cc:301] APPLY: starting
0924 12:46:30.275005 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0924 12:46:30.275046 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0924 12:46:30.275079 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:46:30.275230 (+ 151us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:46:30.275265 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:46:30.275272 (+ 7us) write_op.cc:312] APPLY: finished
0924 12:46:30.275354 (+ 82us) log.cc:842] Serialized 22 byte log entry
0924 12:46:30.276320 (+ 966us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:46:30.276340 (+ 20us) write_op.cc:454] Released schema lock
0924 12:46:30.276348 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":613,"num_ops":1,"prepare.queue_time_us":60,"prepare.run_cpu_time_us":870,"prepare.run_wall_time_us":647895,"replication_time_us":534588,"thread_start_us":545,"threads_started":1}]]}
I20260924 12:46:30.277231 7273 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:48094 (request call id 19) took 1034 ms. Trace:
I20260924 12:46:30.277348 7273 rpcz_store.cc:276] 0924 12:46:29.242915 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:29.242987 (+ 72us) service_pool.cc:227] Handling call
0924 12:46:29.243193 (+ 206us) tablet_service.cc:2780] Created scanner 9b88e7dad98b41428d9892f09dc2242a for tablet 6423621b78684850a4cb8a6a75a63996, query id is 2dc58dff2d2545db87a480c3f43f11b6
0924 12:46:29.243967 (+ 774us) tablet_service.cc:2920] Creating iterator
0924 12:46:29.244000 (+ 33us) tablet_service.cc:3297] Waiting safe time to advance
0924 12:46:29.739911 (+495911us) tablet_service.cc:3304] Waiting for operations to commit
0924 12:46:30.275481 (+535570us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1031468 microseconds
0924 12:46:30.275591 (+ 110us) tablet_service.cc:2945] Iterator created
0924 12:46:30.276091 (+ 500us) tablet_service.cc:2967] Iterator init: OK
0924 12:46:30.276132 (+ 41us) tablet_service.cc:3010] has_more: true
0924 12:46:30.276168 (+ 36us) tablet_service.cc:3027] Continuing scan request
0924 12:46:30.276243 (+ 75us) tablet_service.cc:3091] Found scanner 9b88e7dad98b41428d9892f09dc2242a for tablet 6423621b78684850a4cb8a6a75a63996, query id is 2dc58dff2d2545db87a480c3f43f11b6
0924 12:46:30.277168 (+ 925us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
I20260924 12:46:30.278234 7272 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:48094 (request call id 20) took 1034 ms. Trace:
W20260924 12:46:30.278302 7408 scanner-internal.cc:458] Time spent opening tablet: real 1.036s user 0.001s sys 0.000s
I20260924 12:46:30.278332 7272 rpcz_store.cc:276] 0924 12:46:29.243536 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:29.243630 (+ 94us) service_pool.cc:227] Handling call
0924 12:46:29.243879 (+ 249us) tablet_service.cc:2780] Created scanner dbb2d53560bd4f4d8702daf00ea5fb03 for tablet 6423621b78684850a4cb8a6a75a63996, query id is dc5a796273d4456bb3b1c07bc209a14a
0924 12:46:29.244555 (+ 676us) tablet_service.cc:2920] Creating iterator
0924 12:46:29.244584 (+ 29us) tablet_service.cc:3297] Waiting safe time to advance
0924 12:46:29.739932 (+495348us) tablet_service.cc:3304] Waiting for operations to commit
0924 12:46:30.276757 (+536825us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1032159 microseconds
0924 12:46:30.276840 (+ 83us) tablet_service.cc:2945] Iterator created
0924 12:46:30.277270 (+ 430us) tablet_service.cc:2967] Iterator init: OK
0924 12:46:30.277299 (+ 29us) tablet_service.cc:3010] has_more: true
0924 12:46:30.277330 (+ 31us) tablet_service.cc:3027] Continuing scan request
0924 12:46:30.277405 (+ 75us) tablet_service.cc:3091] Found scanner dbb2d53560bd4f4d8702daf00ea5fb03 for tablet 6423621b78684850a4cb8a6a75a63996, query id is dc5a796273d4456bb3b1c07bc209a14a
0924 12:46:30.278183 (+ 778us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
W20260924 12:46:30.279155 7407 scanner-internal.cc:458] Time spent opening tablet: real 1.036s user 0.001s sys 0.000s
I20260924 12:46:30.512276 7423 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:31.119940 7423 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:31.120136 7423 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:31.120831 7423 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:31.121416 7183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:31.121449 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:31.122345 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9db3b3e723eb41299037b5eab1b00fde; no voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e
I20260924 12:46:31.122795 7423 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:31.123028 7462 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48094 (ReqId={client: b34b87926d454675a830c7cf36b10cff, seq_no=9, attempt_no=0}) took 1366 ms. Trace:
I20260924 12:46:31.123183 7462 rpcz_store.cc:276] 0924 12:46:29.756878 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:29.756940 (+ 62us) service_pool.cc:227] Handling call
0924 12:46:31.122969 (+1366029us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:46:29.757449 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:29.757543 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:29.757548 (+ 5us) write_op.cc:435] Acquired schema lock
0924 12:46:29.757549 (+ 1us) tablet.cc:660] Decoding operations
0924 12:46:29.757627 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:29.757635 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:29.757638 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:29.757677 (+ 39us) tablet.cc:699] Row locks acquired
0924 12:46:29.757678 (+ 1us) write_op.cc:260] PREPARE: finished
0924 12:46:29.757712 (+ 34us) write_op.cc:270] Start()
0924 12:46:30.576857 (+819145us) write_op.cc:276] Timestamp: P: 1790253989757669 usec, L: 0
0924 12:46:30.576863 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:46:30.577126 (+ 263us) log.cc:842] Serialized 354 byte log entry
0924 12:46:31.121532 (+544406us) op_driver.cc:444] REPLICATION: finished
0924 12:46:31.122169 (+ 637us) write_op.cc:301] APPLY: starting
0924 12:46:31.122188 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0924 12:46:31.122230 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0924 12:46:31.122257 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:46:31.122350 (+ 93us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:46:31.122385 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:46:31.122391 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:46:31.122499 (+ 108us) log.cc:842] Serialized 22 byte log entry
0924 12:46:31.122839 (+ 340us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:46:31.122859 (+ 20us) write_op.cc:454] Released schema lock
0924 12:46:31.122866 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":579,"num_ops":1,"prepare.queue_time_us":91,"prepare.run_cpu_time_us":681,"prepare.run_wall_time_us":819747,"replication_time_us":544641,"thread_start_us":462,"threads_started":1}]]}
W20260924 12:46:31.125449 7407 scanner-internal.cc:458] Time spent opening tablet: real 0.696s user 0.001s sys 0.000s
W20260924 12:46:31.125464 7408 scanner-internal.cc:458] Time spent opening tablet: real 0.696s user 0.001s sys 0.000s
I20260924 12:46:31.308990 7421 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:31.309181 7421 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:31.309891 7421 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:31.310410 7051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:31.310734 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:31.311307 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:31.311686 7421 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:31.659351 7423 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:31.659565 7423 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:31.660271 7423 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:31.660818 7183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 13 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:31.660997 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 13 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:31.661738 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9db3b3e723eb41299037b5eab1b00fde; no voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e
I20260924 12:46:31.662110 7423 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:31.662797 7469 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48094 (ReqId={client: b34b87926d454675a830c7cf36b10cff, seq_no=10, attempt_no=0}) took 1384 ms. Trace:
I20260924 12:46:31.662915 7469 rpcz_store.cc:276] 0924 12:46:30.278703 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:30.278747 (+ 44us) service_pool.cc:227] Handling call
0924 12:46:31.662738 (+1383991us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:46:30.577233 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:30.577337 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:30.577342 (+ 5us) write_op.cc:435] Acquired schema lock
0924 12:46:30.577360 (+ 18us) tablet.cc:660] Decoding operations
0924 12:46:30.577436 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:30.577445 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:30.577448 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:30.577509 (+ 61us) tablet.cc:699] Row locks acquired
0924 12:46:30.577511 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:46:30.577554 (+ 43us) write_op.cc:270] Start()
0924 12:46:31.389690 (+812136us) write_op.cc:276] Timestamp: P: 1790253990577521 usec, L: 0
0924 12:46:31.389695 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:46:31.389965 (+ 270us) log.cc:842] Serialized 354 byte log entry
0924 12:46:31.660219 (+270254us) op_driver.cc:444] REPLICATION: finished
0924 12:46:31.662175 (+ 1956us) write_op.cc:301] APPLY: starting
0924 12:46:31.662191 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0924 12:46:31.662335 (+ 144us) tablet.cc:1368] finished BulkCheckPresence
0924 12:46:31.662356 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:46:31.662406 (+ 50us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:46:31.662429 (+ 23us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:46:31.662433 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:46:31.662530 (+ 97us) log.cc:842] Serialized 22 byte log entry
0924 12:46:31.662629 (+ 99us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:46:31.662644 (+ 15us) write_op.cc:454] Released schema lock
0924 12:46:31.662651 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1894,"num_ops":1,"prepare.queue_time_us":298096,"prepare.run_cpu_time_us":752,"prepare.run_wall_time_us":812806,"replication_time_us":270495,"thread_start_us":1897,"threads_started":1}]]}
I20260924 12:46:31.955896 7421 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:31.956089 7421 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:31.956869 7421 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:31.957382 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 13 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:31.957415 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 13 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:31.958029 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:31.992570 7423 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:31.992789 7423 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:31.993652 7423 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:31.994235 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 14 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:31.994447 7183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 14 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:31.995016 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9db3b3e723eb41299037b5eab1b00fde; no voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e
I20260924 12:46:31.995347 7423 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:32.521211 7423 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:32.521427 7423 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:32.522080 7423 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:32.522701 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:32.522689 7183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:32.523366 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9db3b3e723eb41299037b5eab1b00fde; no voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e
W20260924 12:46:32.526022 7408 scanner-internal.cc:458] Time spent opening tablet: real 0.710s user 0.001s sys 0.000s
W20260924 12:46:32.526039 7407 scanner-internal.cc:458] Time spent opening tablet: real 0.710s user 0.001s sys 0.000s
I20260924 12:46:32.544305 7421 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:33.111755 7423 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:33.114174 7483 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48094 (ReqId={client: b34b87926d454675a830c7cf36b10cff, seq_no=13, attempt_no=0}) took 1118 ms. Trace:
I20260924 12:46:33.114298 7483 rpcz_store.cc:276] 0924 12:46:31.995884 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:31.995944 (+ 60us) service_pool.cc:227] Handling call
0924 12:46:33.114120 (+1118176us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:46:32.080768 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:32.080915 (+ 147us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:32.080921 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:46:32.080923 (+ 2us) tablet.cc:660] Decoding operations
0924 12:46:32.081017 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:32.081029 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:32.081035 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:32.081090 (+ 55us) tablet.cc:699] Row locks acquired
0924 12:46:32.081092 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:46:32.081137 (+ 45us) write_op.cc:270] Start()
0924 12:46:32.323283 (+242146us) write_op.cc:276] Timestamp: P: 1790253992081113 usec, L: 0
0924 12:46:32.323288 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:46:32.323605 (+ 317us) log.cc:842] Serialized 354 byte log entry
0924 12:46:33.113279 (+789674us) op_driver.cc:444] REPLICATION: finished
0924 12:46:33.113655 (+ 376us) write_op.cc:301] APPLY: starting
0924 12:46:33.113671 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0924 12:46:33.113711 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0924 12:46:33.113744 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:46:33.113817 (+ 73us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:46:33.113844 (+ 27us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:46:33.113848 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:46:33.113929 (+ 81us) log.cc:842] Serialized 22 byte log entry
0924 12:46:33.114013 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:46:33.114031 (+ 18us) write_op.cc:454] Released schema lock
0924 12:46:33.114037 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":323,"num_ops":1,"prepare.queue_time_us":84426,"prepare.run_cpu_time_us":859,"prepare.run_wall_time_us":242917,"replication_time_us":789966,"thread_start_us":228,"threads_started":1}]]}
I20260924 12:46:33.231503 7421 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:33.231705 7421 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:33.232443 7421 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:33.232909 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 16 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:33.232963 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 16 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:33.233582 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:33.356968 7421 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 12:46:33.360885 7408 scanner-internal.cc:458] Time spent opening tablet: real 0.684s user 0.001s sys 0.000s
W20260924 12:46:33.361068 7407 scanner-internal.cc:458] Time spent opening tablet: real 0.684s user 0.001s sys 0.000s
I20260924 12:46:33.754688 7423 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:33.754885 7423 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:33.755636 7423 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:33.756141 7183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 17 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:33.756228 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 17 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:33.756852 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9db3b3e723eb41299037b5eab1b00fde; no voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e
I20260924 12:46:33.797132 7421 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:33.797330 7421 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:33.798051 7421 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:33.798619 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:33.799297 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:33.800995 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:33.801280 7421 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:34.193796 7423 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:34.844436 7423 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:34.844651 7423 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:34.845314 7423 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:34.845904 7183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 19 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:34.845938 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 19 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:34.846598 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9db3b3e723eb41299037b5eab1b00fde; no voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e
I20260924 12:46:34.847069 7423 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:34.847792 7492 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48094 (ReqId={client: b34b87926d454675a830c7cf36b10cff, seq_no=16, attempt_no=0}) took 1487 ms. Trace:
I20260924 12:46:34.847992 7492 rpcz_store.cc:276] 0924 12:46:33.360523 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:33.360558 (+ 35us) service_pool.cc:227] Handling call
0924 12:46:34.847738 (+1487180us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:46:33.520499 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:33.520630 (+ 131us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:33.520636 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:46:33.520639 (+ 3us) tablet.cc:660] Decoding operations
0924 12:46:33.520733 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:33.520745 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:33.520750 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:33.520810 (+ 60us) tablet.cc:699] Row locks acquired
0924 12:46:33.520811 (+ 1us) write_op.cc:260] PREPARE: finished
0924 12:46:33.520846 (+ 35us) write_op.cc:270] Start()
0924 12:46:34.334005 (+813159us) write_op.cc:276] Timestamp: P: 1790253993520820 usec, L: 0
0924 12:46:34.334010 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:46:34.334299 (+ 289us) log.cc:842] Serialized 354 byte log entry
0924 12:46:34.846493 (+512194us) op_driver.cc:444] REPLICATION: finished
0924 12:46:34.847047 (+ 554us) write_op.cc:301] APPLY: starting
0924 12:46:34.847068 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0924 12:46:34.847120 (+ 52us) tablet.cc:1368] finished BulkCheckPresence
0924 12:46:34.847177 (+ 57us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:46:34.847291 (+ 114us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:46:34.847318 (+ 27us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:46:34.847323 (+ 5us) write_op.cc:312] APPLY: finished
0924 12:46:34.847400 (+ 77us) log.cc:842] Serialized 22 byte log entry
0924 12:46:34.847576 (+ 176us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:46:34.847595 (+ 19us) write_op.cc:454] Released schema lock
0924 12:46:34.847603 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":490,"num_ops":1,"prepare.queue_time_us":159455,"prepare.run_cpu_time_us":868,"prepare.run_wall_time_us":813922,"raft.queue_time_us":37,"raft.run_cpu_time_us":269,"raft.run_wall_time_us":269,"replication_time_us":512425,"spinlock_wait_cycles":51968,"thread_start_us":371,"threads_started":1}]]}
W20260924 12:46:34.849969 7407 scanner-internal.cc:458] Time spent opening tablet: real 0.898s user 0.001s sys 0.000s
W20260924 12:46:34.849992 7408 scanner-internal.cc:458] Time spent opening tablet: real 0.898s user 0.001s sys 0.000s
I20260924 12:46:35.213807 7488 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:35.214097 7488 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:35.214946 7488 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:35.215485 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 19 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:35.216073 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 19 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:35.216858 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:35.565445 7488 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:35.567759 7500 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48094 (ReqId={client: b34b87926d454675a830c7cf36b10cff, seq_no=17, attempt_no=0}) took 1766 ms. Trace:
I20260924 12:46:35.567909 7500 rpcz_store.cc:276] 0924 12:46:33.801344 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:33.801413 (+ 69us) service_pool.cc:227] Handling call
0924 12:46:35.567681 (+1766268us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:46:34.334433 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:34.335063 (+ 630us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:34.335069 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:46:34.335070 (+ 1us) tablet.cc:660] Decoding operations
0924 12:46:34.335149 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:34.335159 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:34.335162 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:34.335205 (+ 43us) tablet.cc:699] Row locks acquired
0924 12:46:34.335206 (+ 1us) write_op.cc:260] PREPARE: finished
0924 12:46:34.335243 (+ 37us) write_op.cc:270] Start()
0924 12:46:34.348379 (+ 13136us) write_op.cc:276] Timestamp: P: 1790253994335191 usec, L: 0
0924 12:46:34.348384 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:46:34.348615 (+ 231us) log.cc:842] Serialized 354 byte log entry
0924 12:46:35.566735 (+1218120us) op_driver.cc:444] REPLICATION: finished
0924 12:46:35.567173 (+ 438us) write_op.cc:301] APPLY: starting
0924 12:46:35.567194 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0924 12:46:35.567230 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0924 12:46:35.567257 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:46:35.567350 (+ 93us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:46:35.567384 (+ 34us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:46:35.567390 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:46:35.567477 (+ 87us) log.cc:842] Serialized 22 byte log entry
0924 12:46:35.567564 (+ 87us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:46:35.567588 (+ 24us) write_op.cc:454] Released schema lock
0924 12:46:35.567595 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":373,"num_ops":1,"prepare.queue_time_us":532528,"prepare.run_cpu_time_us":756,"prepare.run_wall_time_us":14252,"replication_time_us":1218317,"thread_start_us":306,"threads_started":1}]]}
I20260924 12:46:35.593919 7423 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:35.594138 7423 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:35.594903 7423 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:35.595427 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 20 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:35.595460 7183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 20 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:35.596082 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9db3b3e723eb41299037b5eab1b00fde; no voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e
I20260924 12:46:35.874678 7488 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:35.874903 7488 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:35.875631 7488 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:35.876267 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 21 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:35.876295 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 21 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:35.877038 7500 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48094 (ReqId={client: b34b87926d454675a830c7cf36b10cff, seq_no=18, attempt_no=0}) took 1026 ms. Trace:
I20260924 12:46:35.877488 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:35.877295 7500 rpcz_store.cc:276] 0924 12:46:34.850265 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:34.850321 (+ 56us) service_pool.cc:227] Handling call
0924 12:46:35.876970 (+1026649us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:46:34.851043 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:34.851129 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:34.851134 (+ 5us) write_op.cc:435] Acquired schema lock
0924 12:46:34.851135 (+ 1us) tablet.cc:660] Decoding operations
0924 12:46:34.851208 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:34.851218 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:34.851221 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:34.851263 (+ 42us) tablet.cc:699] Row locks acquired
0924 12:46:34.851264 (+ 1us) write_op.cc:260] PREPARE: finished
0924 12:46:34.851296 (+ 32us) write_op.cc:270] Start()
0924 12:46:35.350489 (+499193us) write_op.cc:276] Timestamp: P: 1790253994851254 usec, L: 0
0924 12:46:35.350496 (+ 7us) op_driver.cc:331] REPLICATION: starting
0924 12:46:35.350823 (+ 327us) log.cc:842] Serialized 354 byte log entry
0924 12:46:35.875661 (+524838us) op_driver.cc:444] REPLICATION: finished
0924 12:46:35.875804 (+ 143us) write_op.cc:301] APPLY: starting
0924 12:46:35.875824 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0924 12:46:35.875870 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0924 12:46:35.875910 (+ 40us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:46:35.875980 (+ 70us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:46:35.876008 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:46:35.876012 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:46:35.876112 (+ 100us) log.cc:842] Serialized 22 byte log entry
0924 12:46:35.876804 (+ 692us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:46:35.876829 (+ 25us) write_op.cc:454] Released schema lock
0924 12:46:35.876837 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":70,"num_ops":1,"prepare.queue_time_us":295,"prepare.run_cpu_time_us":787,"prepare.run_wall_time_us":499848,"replication_time_us":525119,"thread_start_us":200,"threads_started":1}]]}
I20260924 12:46:35.877969 7488 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 12:46:35.878903 7408 scanner-internal.cc:458] Time spent opening tablet: real 0.878s user 0.001s sys 0.000s
W20260924 12:46:35.879057 7407 scanner-internal.cc:458] Time spent opening tablet: real 0.878s user 0.001s sys 0.000s
I20260924 12:46:35.937148 7423 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:36.155718 7488 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:36.155947 7488 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:36.156715 7488 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:36.157191 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 22 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:36.157260 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 22 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:36.158018 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:36.158376 7488 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:36.948992 7423 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:36.949201 7423 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:36.949981 7423 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:36.950616 7183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 22 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:36.950634 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 22 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:36.951299 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9db3b3e723eb41299037b5eab1b00fde; no voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e
I20260924 12:46:37.208993 7488 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:37.209203 7488 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:37.209944 7488 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:37.210422 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 23 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:37.210525 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 23 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:37.211228 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:37.211814 7488 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:37.212800 7509 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48094 (ReqId={client: b34b87926d454675a830c7cf36b10cff, seq_no=20, attempt_no=0}) took 1333 ms. Trace:
I20260924 12:46:37.212944 7509 rpcz_store.cc:276] 0924 12:46:35.879399 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:35.879452 (+ 53us) service_pool.cc:227] Handling call
0924 12:46:37.212742 (+1333290us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:46:36.104001 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:36.104126 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:36.104132 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:46:36.104134 (+ 2us) tablet.cc:660] Decoding operations
0924 12:46:36.104230 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:36.104242 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:36.104247 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:36.104329 (+ 82us) tablet.cc:699] Row locks acquired
0924 12:46:36.104331 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:46:36.104377 (+ 46us) write_op.cc:270] Start()
0924 12:46:36.532511 (+428134us) write_op.cc:276] Timestamp: P: 1790253996104354 usec, L: 0
0924 12:46:36.532516 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:46:36.532791 (+ 275us) log.cc:842] Serialized 354 byte log entry
0924 12:46:37.210544 (+677753us) op_driver.cc:444] REPLICATION: finished
0924 12:46:37.212132 (+ 1588us) write_op.cc:301] APPLY: starting
0924 12:46:37.212150 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0924 12:46:37.212191 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0924 12:46:37.212220 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:46:37.212323 (+ 103us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:46:37.212355 (+ 32us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:46:37.212361 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:46:37.212483 (+ 122us) log.cc:842] Serialized 22 byte log entry
0924 12:46:37.212615 (+ 132us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:46:37.212636 (+ 21us) write_op.cc:454] Released schema lock
0924 12:46:37.212645 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1512,"num_ops":1,"prepare.queue_time_us":224207,"prepare.run_cpu_time_us":813,"prepare.run_wall_time_us":428875,"raft.queue_time_us":35,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":677999,"thread_start_us":1378,"threads_started":1}]]}
W20260924 12:46:37.215284 7408 scanner-internal.cc:458] Time spent opening tablet: real 0.905s user 0.001s sys 0.000s
W20260924 12:46:37.215303 7407 scanner-internal.cc:458] Time spent opening tablet: real 0.905s user 0.001s sys 0.000s
I20260924 12:46:37.925464 7423 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:38.302953 7488 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:38.303143 7488 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:38.303864 7488 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:38.304435 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 24 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:38.304520 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 24 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:38.305173 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:38.305438 7488 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:38.306684 7516 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48094 (ReqId={client: b34b87926d454675a830c7cf36b10cff, seq_no=21, attempt_no=0}) took 2146 ms. Trace:
I20260924 12:46:38.306877 7516 rpcz_store.cc:276] 0924 12:46:36.159691 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:36.159996 (+ 305us) service_pool.cc:227] Handling call
0924 12:46:38.306619 (+2146623us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:46:36.532917 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:36.533063 (+ 146us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:36.533069 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:46:36.533071 (+ 2us) tablet.cc:660] Decoding operations
0924 12:46:36.533194 (+ 123us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:36.533206 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:36.533211 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:36.533265 (+ 54us) tablet.cc:699] Row locks acquired
0924 12:46:36.533267 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:46:36.533309 (+ 42us) write_op.cc:270] Start()
0924 12:46:37.437462 (+904153us) write_op.cc:276] Timestamp: P: 1790253996533251 usec, L: 0
0924 12:46:37.437468 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:46:37.437765 (+ 297us) log.cc:842] Serialized 354 byte log entry
0924 12:46:38.305094 (+867329us) op_driver.cc:444] REPLICATION: finished
0924 12:46:38.305517 (+ 423us) write_op.cc:301] APPLY: starting
0924 12:46:38.305539 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0924 12:46:38.305583 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0924 12:46:38.305605 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:46:38.305673 (+ 68us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:46:38.305702 (+ 29us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:46:38.305706 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:46:38.305783 (+ 77us) log.cc:842] Serialized 22 byte log entry
0924 12:46:38.306482 (+ 699us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:46:38.306510 (+ 28us) write_op.cc:454] Released schema lock
0924 12:46:38.306517 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":351,"num_ops":1,"prepare.queue_time_us":372525,"prepare.run_cpu_time_us":845,"prepare.run_wall_time_us":904916,"replication_time_us":867484,"thread_start_us":447,"threads_started":1}]]}
W20260924 12:46:38.308667 7407 scanner-internal.cc:458] Time spent opening tablet: real 0.943s user 0.001s sys 0.000s
W20260924 12:46:38.308694 7408 scanner-internal.cc:458] Time spent opening tablet: real 0.943s user 0.001s sys 0.000s
I20260924 12:46:38.315986 7488 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:38.316169 7488 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:38.316840 7488 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:38.317395 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 25 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:38.317395 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 25 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:38.317728 7516 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48094 (ReqId={client: b34b87926d454675a830c7cf36b10cff, seq_no=22, attempt_no=0}) took 1102 ms. Trace:
I20260924 12:46:38.317855 7516 rpcz_store.cc:276] 0924 12:46:37.215113 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:37.215566 (+ 453us) service_pool.cc:227] Handling call
0924 12:46:38.317672 (+1102106us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:46:37.437863 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:37.437998 (+ 135us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:37.438004 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:46:37.438006 (+ 2us) tablet.cc:660] Decoding operations
0924 12:46:37.438107 (+ 101us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:37.438120 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:37.438124 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:37.438181 (+ 57us) tablet.cc:699] Row locks acquired
0924 12:46:37.438184 (+ 3us) write_op.cc:260] PREPARE: finished
0924 12:46:37.438229 (+ 45us) write_op.cc:270] Start()
0924 12:46:37.680422 (+242193us) write_op.cc:276] Timestamp: P: 1790253997438157 usec, L: 0
0924 12:46:37.680428 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:46:37.680678 (+ 250us) log.cc:842] Serialized 354 byte log entry
0924 12:46:38.316920 (+636242us) op_driver.cc:444] REPLICATION: finished
0924 12:46:38.317040 (+ 120us) write_op.cc:301] APPLY: starting
0924 12:46:38.317057 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0924 12:46:38.317107 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
0924 12:46:38.317158 (+ 51us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:46:38.317225 (+ 67us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:46:38.317259 (+ 34us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:46:38.317264 (+ 5us) write_op.cc:312] APPLY: finished
0924 12:46:38.317462 (+ 198us) log.cc:842] Serialized 22 byte log entry
0924 12:46:38.317559 (+ 97us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:46:38.317576 (+ 17us) write_op.cc:454] Released schema lock
0924 12:46:38.317582 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":73,"num_ops":1,"prepare.queue_time_us":221843,"prepare.run_cpu_time_us":831,"prepare.run_wall_time_us":242896,"replication_time_us":636466}]]}
I20260924 12:46:38.318101 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:38.318341 7488 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:39.016723 7423 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:39.016928 7423 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:39.017132 7051 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.5.77.195:57505 (request call id 1575) took 1090 ms. Trace:
I20260924 12:46:39.017632 7423 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:39.017287 7051 rpcz_store.cc:276] 0924 12:46:37.926878 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:37.926936 (+ 58us) service_pool.cc:227] Handling call
0924 12:46:37.927180 (+ 244us) raft_consensus.cc:1457] Updating replica for 2 ops
0924 12:46:37.927290 (+ 110us) raft_consensus.cc:1520] Early marking committed up to index 23
0924 12:46:37.927544 (+ 254us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0924 12:46:37.927669 (+ 125us) write_op.cc:270] Start()
0924 12:46:38.472807 (+545138us) write_op.cc:276] Timestamp: P: 1790253996533251 usec, L: 0
0924 12:46:38.473402 (+ 595us) write_op.cc:270] Start()
0924 12:46:39.015578 (+542176us) write_op.cc:276] Timestamp: P: 1790253997438157 usec, L: 0
0924 12:46:39.016273 (+ 695us) log.cc:842] Serialized 708 byte log entry
0924 12:46:39.016577 (+ 304us) raft_consensus.cc:1631] Marking committed up to 23
0924 12:46:39.016589 (+ 12us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 12:46:39.016629 (+ 40us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 12:46:39.017032 (+ 403us) raft_consensus.cc:1671] finished
0924 12:46:39.017034 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 12:46:39.017089 (+ 55us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:46:38.473280 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:38.473402 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:38.473408 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:46:38.473410 (+ 2us) tablet.cc:660] Decoding operations
0924 12:46:38.473507 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:38.473524 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:38.473529 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:38.473612 (+ 83us) tablet.cc:699] Row locks acquired
0924 12:46:38.473614 (+ 2us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0924 12:46:39.016093 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:39.016276 (+ 183us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:39.016282 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:46:39.016283 (+ 1us) tablet.cc:660] Decoding operations
0924 12:46:39.016397 (+ 114us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:39.016444 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:39.016450 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:39.016506 (+ 56us) tablet.cc:699] Row locks acquired
0924 12:46:39.016508 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":219,"threads_started":1,"wal-append.queue_time_us":297,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":8,"prepare.queue_time_us":329,"prepare.run_cpu_time_us":454,"prepare.run_wall_time_us":455,"thread_start_us":230,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":36,"prepare.queue_time_us":365,"prepare.run_cpu_time_us":528,"prepare.run_wall_time_us":529,"thread_start_us":257,"threads_started":1}]]}
I20260924 12:46:39.018261 7183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 25 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:39.018296 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 25 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:39.018939 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9db3b3e723eb41299037b5eab1b00fde; no voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e
I20260924 12:46:39.019169 7423 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:39.187784 7488 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:39.187986 7488 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:39.188767 7488 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:39.189349 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 26 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:39.189340 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 26 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:39.190192 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
W20260924 12:46:39.193048 7407 scanner-internal.cc:458] Time spent opening tablet: real 0.734s user 0.001s sys 0.000s
W20260924 12:46:39.193348 7408 scanner-internal.cc:458] Time spent opening tablet: real 0.734s user 0.001s sys 0.000s
I20260924 12:46:39.682195 7423 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:39.682399 7423 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:39.683437 7423 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:39.684067 7527 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48094 (ReqId={client: b34b87926d454675a830c7cf36b10cff, seq_no=24, attempt_no=0}) took 1364 ms. Trace:
I20260924 12:46:39.684191 7183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:39.684461 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:39.684201 7527 rpcz_store.cc:276] 0924 12:46:38.319603 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:38.319675 (+ 72us) service_pool.cc:227] Handling call
0924 12:46:39.684001 (+1364326us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:46:38.466137 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:38.466252 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:38.466258 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:46:38.466261 (+ 3us) tablet.cc:660] Decoding operations
0924 12:46:38.466359 (+ 98us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:38.466380 (+ 21us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:38.466383 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:38.466432 (+ 49us) tablet.cc:699] Row locks acquired
0924 12:46:38.466434 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:46:38.466509 (+ 75us) write_op.cc:270] Start()
0924 12:46:38.854662 (+388153us) write_op.cc:276] Timestamp: P: 1790253998466455 usec, L: 0
0924 12:46:38.854667 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:46:38.854889 (+ 222us) log.cc:842] Serialized 354 byte log entry
0924 12:46:39.683347 (+828458us) op_driver.cc:444] REPLICATION: finished
0924 12:46:39.683474 (+ 127us) write_op.cc:301] APPLY: starting
0924 12:46:39.683492 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0924 12:46:39.683535 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0924 12:46:39.683570 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:46:39.683660 (+ 90us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:46:39.683695 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:46:39.683701 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:46:39.683786 (+ 85us) log.cc:842] Serialized 22 byte log entry
0924 12:46:39.683869 (+ 83us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:46:39.683888 (+ 19us) write_op.cc:454] Released schema lock
0924 12:46:39.683896 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":71,"num_ops":1,"prepare.queue_time_us":146039,"prepare.run_cpu_time_us":787,"prepare.run_wall_time_us":388851,"replication_time_us":828651}]]}
I20260924 12:46:39.685004 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9db3b3e723eb41299037b5eab1b00fde; no voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e
I20260924 12:46:39.686439 7423 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:40.010251 7423 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:40.010493 7423 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:40.011204 7423 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:40.011786 7183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:40.011767 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
W20260924 12:46:40.014205 7407 scanner-internal.cc:458] Time spent opening tablet: real 0.670s user 0.001s sys 0.000s
W20260924 12:46:40.014577 7408 scanner-internal.cc:458] Time spent opening tablet: real 0.671s user 0.000s sys 0.001s
I20260924 12:46:40.014879 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9db3b3e723eb41299037b5eab1b00fde; no voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e
I20260924 12:46:40.015146 7423 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:40.114259 7488 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:40.499163 7488 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:40.499348 7488 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:40.500303 7488 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:40.500605 7051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:40.501120 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:40.501672 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:40.502100 7488 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:41.298108 7534 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:41.298321 7534 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:41.299114 7534 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:41.299659 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:41.299682 7183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:41.300382 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9db3b3e723eb41299037b5eab1b00fde; no voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e
I20260924 12:46:41.300702 7534 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:41.301488 7539 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48094 (ReqId={client: b34b87926d454675a830c7cf36b10cff, seq_no=26, attempt_no=0}) took 1614 ms. Trace:
I20260924 12:46:41.301684 7539 rpcz_store.cc:276] 0924 12:46:39.686719 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:39.686772 (+ 53us) service_pool.cc:227] Handling call
0924 12:46:41.301431 (+1614659us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:46:39.836768 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:39.836869 (+ 101us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:39.836874 (+ 5us) write_op.cc:435] Acquired schema lock
0924 12:46:39.836875 (+ 1us) tablet.cc:660] Decoding operations
0924 12:46:39.836969 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:39.836978 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:39.836982 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:39.837025 (+ 43us) tablet.cc:699] Row locks acquired
0924 12:46:39.837027 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:46:39.837061 (+ 34us) write_op.cc:270] Start()
0924 12:46:40.660237 (+823176us) write_op.cc:276] Timestamp: P: 1790253999837020 usec, L: 0
0924 12:46:40.660243 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:46:40.660553 (+ 310us) log.cc:842] Serialized 354 byte log entry
0924 12:46:41.300318 (+639765us) op_driver.cc:444] REPLICATION: finished
0924 12:46:41.300829 (+ 511us) write_op.cc:301] APPLY: starting
0924 12:46:41.300849 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0924 12:46:41.300894 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0924 12:46:41.300916 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:46:41.301023 (+ 107us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:46:41.301051 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:46:41.301055 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:46:41.301166 (+ 111us) log.cc:842] Serialized 22 byte log entry
0924 12:46:41.301283 (+ 117us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:46:41.301303 (+ 20us) write_op.cc:454] Released schema lock
0924 12:46:41.301310 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":443,"num_ops":1,"prepare.queue_time_us":149596,"prepare.run_cpu_time_us":779,"prepare.run_wall_time_us":823863,"replication_time_us":640050,"thread_start_us":316,"threads_started":1}]]}
I20260924 12:46:41.303431 7272 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:48094 (request call id 65) took 1137 ms. Trace:
I20260924 12:46:41.303546 7272 rpcz_store.cc:276] 0924 12:46:40.165753 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:40.165814 (+ 61us) service_pool.cc:227] Handling call
0924 12:46:40.165934 (+ 120us) tablet_service.cc:2780] Created scanner 585b14c056974da2b9ad4dcdc3033f9c for tablet 6423621b78684850a4cb8a6a75a63996, query id is f9ba032e9f9f44e9a91ac9901d2909c2
0924 12:46:40.166840 (+ 906us) tablet_service.cc:2920] Creating iterator
0924 12:46:40.166892 (+ 52us) tablet_service.cc:3297] Waiting safe time to advance
0924 12:46:40.660378 (+493486us) tablet_service.cc:3304] Waiting for operations to commit
0924 12:46:41.301475 (+641097us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1134567 microseconds
0924 12:46:41.301602 (+ 127us) tablet_service.cc:2945] Iterator created
0924 12:46:41.302196 (+ 594us) tablet_service.cc:2967] Iterator init: OK
0924 12:46:41.302235 (+ 39us) tablet_service.cc:3010] has_more: true
0924 12:46:41.302272 (+ 37us) tablet_service.cc:3027] Continuing scan request
0924 12:46:41.302336 (+ 64us) tablet_service.cc:3091] Found scanner 585b14c056974da2b9ad4dcdc3033f9c for tablet 6423621b78684850a4cb8a6a75a63996, query id is f9ba032e9f9f44e9a91ac9901d2909c2
0924 12:46:41.303331 (+ 995us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":15232}
W20260924 12:46:41.304271 7408 scanner-internal.cc:458] Time spent opening tablet: real 1.139s user 0.000s sys 0.001s
I20260924 12:46:41.304731 7271 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:48094 (request call id 64) took 1139 ms. Trace:
I20260924 12:46:41.305258 7271 rpcz_store.cc:276] 0924 12:46:40.165612 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:40.165669 (+ 57us) service_pool.cc:227] Handling call
0924 12:46:40.165867 (+ 198us) tablet_service.cc:2780] Created scanner 3d087354269f41409736851a2604d69e for tablet 6423621b78684850a4cb8a6a75a63996, query id is 55de708186324bff9ebcb9cd4132ff8c
0924 12:46:40.166847 (+ 980us) tablet_service.cc:2920] Creating iterator
0924 12:46:40.166895 (+ 48us) tablet_service.cc:3297] Waiting safe time to advance
0924 12:46:40.661191 (+494296us) tablet_service.cc:3304] Waiting for operations to commit
0924 12:46:41.301338 (+640147us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1134431 microseconds
0924 12:46:41.301447 (+ 109us) tablet_service.cc:2945] Iterator created
0924 12:46:41.302052 (+ 605us) tablet_service.cc:2967] Iterator init: OK
0924 12:46:41.302085 (+ 33us) tablet_service.cc:3010] has_more: true
0924 12:46:41.302120 (+ 35us) tablet_service.cc:3027] Continuing scan request
0924 12:46:41.302193 (+ 73us) tablet_service.cc:3091] Found scanner 3d087354269f41409736851a2604d69e for tablet 6423621b78684850a4cb8a6a75a63996, query id is 55de708186324bff9ebcb9cd4132ff8c
0924 12:46:41.304690 (+ 2497us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":39040}
W20260924 12:46:41.305989 7407 scanner-internal.cc:458] Time spent opening tablet: real 1.141s user 0.001s sys 0.000s
I20260924 12:46:41.592855 7488 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:41.593060 7488 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:41.593765 7488 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:41.594290 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:41.594419 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:41.595525 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:41.815806 7534 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:41.816092 7534 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:41.816777 7534 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:41.817394 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:41.817394 7183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:41.817993 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9db3b3e723eb41299037b5eab1b00fde; no voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e
I20260924 12:46:41.818732 7546 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48094 (ReqId={client: b34b87926d454675a830c7cf36b10cff, seq_no=27, attempt_no=0}) took 1805 ms. Trace:
I20260924 12:46:41.818874 7546 rpcz_store.cc:276] 0924 12:46:40.013570 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:40.013703 (+ 133us) service_pool.cc:227] Handling call
0924 12:46:41.818671 (+1804968us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:46:40.660679 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:40.660837 (+ 158us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:40.660844 (+ 7us) write_op.cc:435] Acquired schema lock
0924 12:46:40.660847 (+ 3us) tablet.cc:660] Decoding operations
0924 12:46:40.660981 (+ 134us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:40.660994 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:40.661000 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:40.661083 (+ 83us) tablet.cc:699] Row locks acquired
0924 12:46:40.661085 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:46:40.661142 (+ 57us) write_op.cc:270] Start()
0924 12:46:41.369310 (+708168us) write_op.cc:276] Timestamp: P: 1790254000661119 usec, L: 0
0924 12:46:41.369316 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:46:41.369573 (+ 257us) log.cc:842] Serialized 354 byte log entry
0924 12:46:41.817180 (+447607us) op_driver.cc:444] REPLICATION: finished
0924 12:46:41.818107 (+ 927us) write_op.cc:301] APPLY: starting
0924 12:46:41.818127 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0924 12:46:41.818173 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0924 12:46:41.818227 (+ 54us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:46:41.818303 (+ 76us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:46:41.818337 (+ 34us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:46:41.818342 (+ 5us) write_op.cc:312] APPLY: finished
0924 12:46:41.818428 (+ 86us) log.cc:842] Serialized 22 byte log entry
0924 12:46:41.818552 (+ 124us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:46:41.818571 (+ 19us) write_op.cc:454] Released schema lock
0924 12:46:41.818579 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":865,"num_ops":1,"prepare.queue_time_us":646534,"prepare.run_cpu_time_us":892,"prepare.run_wall_time_us":708976,"replication_time_us":447836,"thread_start_us":774,"threads_started":1}]]}
I20260924 12:46:42.025286 7534 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 12:46:42.029273 7408 scanner-internal.cc:458] Time spent opening tablet: real 0.574s user 0.000s sys 0.001s
W20260924 12:46:42.029338 7407 scanner-internal.cc:458] Time spent opening tablet: real 0.573s user 0.001s sys 0.000s
I20260924 12:46:42.043030 7534 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:42.043241 7534 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:42.043953 7534 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:42.044557 7183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:42.044564 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:42.045353 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9db3b3e723eb41299037b5eab1b00fde; no voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e
I20260924 12:46:42.045807 7534 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:42.437228 7488 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:42.550135 7488 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:42.550323 7488 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:42.551090 7488 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:42.551533 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:42.551517 7051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:42.552062 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:42.552270 7488 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:43.257192 7551 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:43.257409 7551 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:43.258241 7551 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:43.258790 7183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:43.259692 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:43.260226 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9db3b3e723eb41299037b5eab1b00fde; no voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e
I20260924 12:46:43.260417 7554 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48094 (ReqId={client: b34b87926d454675a830c7cf36b10cff, seq_no=30, attempt_no=0}) took 1231 ms. Trace:
I20260924 12:46:43.261023 7551 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:43.260593 7554 rpcz_store.cc:276] 0924 12:46:42.029407 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:42.029585 (+ 178us) service_pool.cc:227] Handling call
0924 12:46:43.260367 (+1230782us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:46:42.030006 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:42.030091 (+ 85us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:42.030096 (+ 5us) write_op.cc:435] Acquired schema lock
0924 12:46:42.030098 (+ 2us) tablet.cc:660] Decoding operations
0924 12:46:42.030158 (+ 60us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:42.030166 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:42.030169 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:42.030207 (+ 38us) tablet.cc:699] Row locks acquired
0924 12:46:42.030208 (+ 1us) write_op.cc:260] PREPARE: finished
0924 12:46:42.030237 (+ 29us) write_op.cc:270] Start()
0924 12:46:42.790381 (+760144us) write_op.cc:276] Timestamp: P: 1790254002030189 usec, L: 0
0924 12:46:42.790386 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:46:42.790700 (+ 314us) log.cc:842] Serialized 354 byte log entry
0924 12:46:43.258227 (+467527us) op_driver.cc:444] REPLICATION: finished
0924 12:46:43.259737 (+ 1510us) write_op.cc:301] APPLY: starting
0924 12:46:43.259754 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0924 12:46:43.259791 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0924 12:46:43.259821 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:46:43.259928 (+ 107us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:46:43.259958 (+ 30us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:46:43.259967 (+ 9us) write_op.cc:312] APPLY: finished
0924 12:46:43.260112 (+ 145us) log.cc:842] Serialized 22 byte log entry
0924 12:46:43.260249 (+ 137us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:46:43.260267 (+ 18us) write_op.cc:454] Released schema lock
0924 12:46:43.260274 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1446,"num_ops":1,"prepare.queue_time_us":56,"prepare.run_cpu_time_us":696,"prepare.run_wall_time_us":760765,"replication_time_us":467811,"thread_start_us":516,"threads_started":1}]]}
I20260924 12:46:43.262375 7272 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:48094 (request call id 72) took 1081 ms. Trace:
I20260924 12:46:43.262544 7272 rpcz_store.cc:276] 0924 12:46:42.180679 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:42.180785 (+ 106us) service_pool.cc:227] Handling call
0924 12:46:42.181027 (+ 242us) tablet_service.cc:2780] Created scanner 682473e8aecc4ab59a56bf59f0505741 for tablet 6423621b78684850a4cb8a6a75a63996, query id is 62de666c088645658eb317f13b784b41
0924 12:46:42.181816 (+ 789us) tablet_service.cc:2920] Creating iterator
0924 12:46:42.181866 (+ 50us) tablet_service.cc:3297] Waiting safe time to advance
0924 12:46:42.791248 (+609382us) tablet_service.cc:3304] Waiting for operations to commit
0924 12:46:43.260465 (+469217us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1078583 microseconds
0924 12:46:43.260589 (+ 124us) tablet_service.cc:2945] Iterator created
0924 12:46:43.261197 (+ 608us) tablet_service.cc:2967] Iterator init: OK
0924 12:46:43.261237 (+ 40us) tablet_service.cc:3010] has_more: true
0924 12:46:43.261277 (+ 40us) tablet_service.cc:3027] Continuing scan request
0924 12:46:43.261361 (+ 84us) tablet_service.cc:3091] Found scanner 682473e8aecc4ab59a56bf59f0505741 for tablet 6423621b78684850a4cb8a6a75a63996, query id is 62de666c088645658eb317f13b784b41
0924 12:46:43.262304 (+ 943us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":28928}
I20260924 12:46:43.263150 7271 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:48094 (request call id 73) took 1082 ms. Trace:
I20260924 12:46:43.263258 7271 rpcz_store.cc:276] 0924 12:46:42.180810 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:42.180858 (+ 48us) service_pool.cc:227] Handling call
0924 12:46:42.181026 (+ 168us) tablet_service.cc:2780] Created scanner 7601bfdbe70046b3bb74d2086a5792d2 for tablet 6423621b78684850a4cb8a6a75a63996, query id is d28c4cbec89d441796b459da8331254d
0924 12:46:42.181811 (+ 785us) tablet_service.cc:2920] Creating iterator
0924 12:46:42.181864 (+ 53us) tablet_service.cc:3297] Waiting safe time to advance
0924 12:46:42.791216 (+609352us) tablet_service.cc:3304] Waiting for operations to commit
0924 12:46:43.260516 (+469300us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1078641 microseconds
0924 12:46:43.260600 (+ 84us) tablet_service.cc:2945] Iterator created
0924 12:46:43.261208 (+ 608us) tablet_service.cc:2967] Iterator init: OK
0924 12:46:43.261245 (+ 37us) tablet_service.cc:3010] has_more: true
0924 12:46:43.261277 (+ 32us) tablet_service.cc:3027] Continuing scan request
0924 12:46:43.261361 (+ 84us) tablet_service.cc:3091] Found scanner 7601bfdbe70046b3bb74d2086a5792d2 for tablet 6423621b78684850a4cb8a6a75a63996, query id is d28c4cbec89d441796b459da8331254d
0924 12:46:43.263086 (+ 1725us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":51712}
W20260924 12:46:43.263760 7408 scanner-internal.cc:458] Time spent opening tablet: real 1.084s user 0.000s sys 0.001s
W20260924 12:46:43.264277 7407 scanner-internal.cc:458] Time spent opening tablet: real 1.084s user 0.001s sys 0.000s
I20260924 12:46:43.264757 7554 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48094 (ReqId={client: b34b87926d454675a830c7cf36b10cff, seq_no=31, attempt_no=0}) took 1218 ms. Trace:
I20260924 12:46:43.264853 7554 rpcz_store.cc:276] 0924 12:46:42.046559 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:42.046624 (+ 65us) service_pool.cc:227] Handling call
0924 12:46:43.264705 (+1218081us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:46:42.790816 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:42.790948 (+ 132us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:42.790954 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:46:42.790956 (+ 2us) tablet.cc:660] Decoding operations
0924 12:46:42.791053 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:42.791064 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:42.791069 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:42.791126 (+ 57us) tablet.cc:699] Row locks acquired
0924 12:46:42.791129 (+ 3us) write_op.cc:260] PREPARE: finished
0924 12:46:42.791174 (+ 45us) write_op.cc:270] Start()
0924 12:46:43.030327 (+239153us) write_op.cc:276] Timestamp: P: 1790254002791143 usec, L: 0
0924 12:46:43.030332 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:46:43.030585 (+ 253us) log.cc:842] Serialized 354 byte log entry
0924 12:46:43.262780 (+232195us) op_driver.cc:444] REPLICATION: finished
0924 12:46:43.263359 (+ 579us) write_op.cc:301] APPLY: starting
0924 12:46:43.263376 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0924 12:46:43.263411 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0924 12:46:43.263454 (+ 43us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:46:43.263516 (+ 62us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:46:43.263547 (+ 31us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:46:43.263553 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:46:43.264329 (+ 776us) log.cc:842] Serialized 22 byte log entry
0924 12:46:43.264596 (+ 267us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:46:43.264633 (+ 37us) write_op.cc:454] Released schema lock
0924 12:46:43.264639 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":519,"num_ops":1,"prepare.queue_time_us":743729,"prepare.run_cpu_time_us":790,"prepare.run_wall_time_us":239853,"replication_time_us":232426}]]}
I20260924 12:46:43.338716 7488 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:43.338901 7488 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:43.339547 7488 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:43.340154 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:43.340240 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:43.340878 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:44.064429 7488 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:44.394230 7488 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:44.394501 7488 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:44.395269 7488 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:44.396060 7051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 35 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:44.396255 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 35 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:44.396692 7562 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48094 (ReqId={client: b34b87926d454675a830c7cf36b10cff, seq_no=32, attempt_no=0}) took 1132 ms. Trace:
I20260924 12:46:44.396886 7562 rpcz_store.cc:276] 0924 12:46:43.263947 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:43.263997 (+ 50us) service_pool.cc:227] Handling call
0924 12:46:44.396630 (+1132633us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:46:43.264521 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:43.264628 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:43.264632 (+ 4us) write_op.cc:435] Acquired schema lock
0924 12:46:43.264634 (+ 2us) tablet.cc:660] Decoding operations
0924 12:46:43.264700 (+ 66us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:43.264709 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:43.264712 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:43.264751 (+ 39us) tablet.cc:699] Row locks acquired
0924 12:46:43.264753 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:46:43.264783 (+ 30us) write_op.cc:270] Start()
0924 12:46:44.105931 (+841148us) write_op.cc:276] Timestamp: P: 1790254003264750 usec, L: 0
0924 12:46:44.105936 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:46:44.106220 (+ 284us) log.cc:842] Serialized 354 byte log entry
0924 12:46:44.395380 (+289160us) op_driver.cc:444] REPLICATION: finished
0924 12:46:44.395991 (+ 611us) write_op.cc:301] APPLY: starting
0924 12:46:44.396010 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0924 12:46:44.396055 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0924 12:46:44.396083 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:46:44.396198 (+ 115us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:46:44.396235 (+ 37us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:46:44.396245 (+ 10us) write_op.cc:312] APPLY: finished
0924 12:46:44.396367 (+ 122us) log.cc:842] Serialized 22 byte log entry
0924 12:46:44.396489 (+ 122us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:46:44.396507 (+ 18us) write_op.cc:454] Released schema lock
0924 12:46:44.396514 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":540,"num_ops":1,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":734,"prepare.run_wall_time_us":841796,"replication_time_us":289416,"thread_start_us":1479,"threads_started":1}]]}
I20260924 12:46:44.399655 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
W20260924 12:46:44.399878 7408 scanner-internal.cc:458] Time spent opening tablet: real 0.985s user 0.001s sys 0.000s
W20260924 12:46:44.400204 7407 scanner-internal.cc:458] Time spent opening tablet: real 0.985s user 0.001s sys 0.000s
I20260924 12:46:44.766114 7561 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:44.766377 7561 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:44.767261 7561 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:44.767915 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 35 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:44.767940 7183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 35 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:44.768615 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9db3b3e723eb41299037b5eab1b00fde; no voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e
I20260924 12:46:45.169183 7488 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:45.171676 7569 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48094 (ReqId={client: b34b87926d454675a830c7cf36b10cff, seq_no=33, attempt_no=0}) took 1905 ms. Trace:
I20260924 12:46:45.171825 7569 rpcz_store.cc:276] 0924 12:46:43.266415 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:43.266506 (+ 91us) service_pool.cc:227] Handling call
0924 12:46:45.171620 (+1905114us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:46:44.106340 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:44.106482 (+ 142us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:44.106489 (+ 7us) write_op.cc:435] Acquired schema lock
0924 12:46:44.106491 (+ 2us) tablet.cc:660] Decoding operations
0924 12:46:44.106591 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:44.106603 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:44.106608 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:44.106665 (+ 57us) tablet.cc:699] Row locks acquired
0924 12:46:44.106667 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:46:44.106717 (+ 50us) write_op.cc:270] Start()
0924 12:46:44.138868 (+ 32151us) write_op.cc:276] Timestamp: P: 1790254004106672 usec, L: 0
0924 12:46:44.138873 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:46:44.139099 (+ 226us) log.cc:842] Serialized 354 byte log entry
0924 12:46:45.170355 (+1031256us) op_driver.cc:444] REPLICATION: finished
0924 12:46:45.171037 (+ 682us) write_op.cc:301] APPLY: starting
0924 12:46:45.171057 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0924 12:46:45.171098 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0924 12:46:45.171122 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:46:45.171254 (+ 132us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:46:45.171289 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:46:45.171295 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:46:45.171399 (+ 104us) log.cc:842] Serialized 22 byte log entry
0924 12:46:45.171486 (+ 87us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:46:45.171508 (+ 22us) write_op.cc:454] Released schema lock
0924 12:46:45.171516 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":620,"num_ops":1,"prepare.queue_time_us":839470,"prepare.run_cpu_time_us":765,"prepare.run_wall_time_us":32829,"replication_time_us":1031453,"thread_start_us":514,"threads_started":1}]]}
I20260924 12:46:45.233569 7488 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:45.233777 7488 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:45.234558 7488 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:45.235211 7051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 37 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:45.235983 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 37 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:45.236822 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:45.237164 7488 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 12:46:45.237619 7407 scanner-internal.cc:458] Time spent opening tablet: real 0.687s user 0.000s sys 0.001s
W20260924 12:46:45.237884 7408 scanner-internal.cc:458] Time spent opening tablet: real 0.687s user 0.001s sys 0.000s
I20260924 12:46:45.557580 7561 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:45.765496 7488 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:45.765713 7488 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:45.766476 7488 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:45.767038 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 38 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:45.767057 7051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 38 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:45.767772 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:45.768762 7488 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:46.529537 7561 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:46.529762 7561 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:46.530429 7561 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:46.531029 7183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 38 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:46.531023 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 38 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:46.531738 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9db3b3e723eb41299037b5eab1b00fde; no voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e
I20260924 12:46:47.350059 7561 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:47.352542 7585 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48094 (ReqId={client: b34b87926d454675a830c7cf36b10cff, seq_no=36, attempt_no=0}) took 2113 ms. Trace:
I20260924 12:46:47.352778 7585 rpcz_store.cc:276] 0924 12:46:45.238784 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:45.238846 (+ 62us) service_pool.cc:227] Handling call
0924 12:46:47.352474 (+2113628us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:46:45.682479 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:45.682603 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:45.682609 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:46:45.682611 (+ 2us) tablet.cc:660] Decoding operations
0924 12:46:45.682715 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:45.682725 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:45.682728 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:45.682772 (+ 44us) tablet.cc:699] Row locks acquired
0924 12:46:45.682773 (+ 1us) write_op.cc:260] PREPARE: finished
0924 12:46:45.682808 (+ 35us) write_op.cc:270] Start()
0924 12:46:46.496966 (+814158us) write_op.cc:276] Timestamp: P: 1790254005682755 usec, L: 0
0924 12:46:46.496971 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:46:46.497237 (+ 266us) log.cc:842] Serialized 354 byte log entry
0924 12:46:47.351434 (+854197us) op_driver.cc:444] REPLICATION: finished
0924 12:46:47.351857 (+ 423us) write_op.cc:301] APPLY: starting
0924 12:46:47.351875 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0924 12:46:47.351918 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0924 12:46:47.351968 (+ 50us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:46:47.352077 (+ 109us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:46:47.352106 (+ 29us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:46:47.352111 (+ 5us) write_op.cc:312] APPLY: finished
0924 12:46:47.352202 (+ 91us) log.cc:842] Serialized 22 byte log entry
0924 12:46:47.352332 (+ 130us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:46:47.352351 (+ 19us) write_op.cc:454] Released schema lock
0924 12:46:47.352358 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":366,"num_ops":1,"prepare.queue_time_us":443190,"prepare.run_cpu_time_us":788,"prepare.run_wall_time_us":814869,"raft.queue_time_us":35,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":208,"replication_time_us":854440,"thread_start_us":253,"threads_started":1}]]}
I20260924 12:46:47.354363 7271 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:48094 (request call id 87) took 1432 ms. Trace:
I20260924 12:46:47.354487 7271 rpcz_store.cc:276] 0924 12:46:45.922048 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:45.922101 (+ 53us) service_pool.cc:227] Handling call
0924 12:46:45.922220 (+ 119us) tablet_service.cc:2780] Created scanner d4273457e92743aca08808f4a3d2e7d5 for tablet 6423621b78684850a4cb8a6a75a63996, query id is d6bb9370b1154b36b9955f6d1e2a02a7
0924 12:46:45.923712 (+ 1492us) tablet_service.cc:2920] Creating iterator
0924 12:46:45.923781 (+ 69us) tablet_service.cc:3297] Waiting safe time to advance
0924 12:46:46.497120 (+573339us) tablet_service.cc:3304] Waiting for operations to commit
0924 12:46:47.352459 (+855339us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1428667 microseconds
0924 12:46:47.352573 (+ 114us) tablet_service.cc:2945] Iterator created
0924 12:46:47.353158 (+ 585us) tablet_service.cc:2967] Iterator init: OK
0924 12:46:47.353192 (+ 34us) tablet_service.cc:3010] has_more: true
0924 12:46:47.353228 (+ 36us) tablet_service.cc:3027] Continuing scan request
0924 12:46:47.353343 (+ 115us) tablet_service.cc:3091] Found scanner d4273457e92743aca08808f4a3d2e7d5 for tablet 6423621b78684850a4cb8a6a75a63996, query id is d6bb9370b1154b36b9955f6d1e2a02a7
0924 12:46:47.354295 (+ 952us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":41984}
I20260924 12:46:47.354790 7272 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:48094 (request call id 86) took 1432 ms. Trace:
I20260924 12:46:47.354902 7272 rpcz_store.cc:276] 0924 12:46:45.921861 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:45.921949 (+ 88us) service_pool.cc:227] Handling call
0924 12:46:45.922189 (+ 240us) tablet_service.cc:2780] Created scanner 086ad9685825406892f2027b34b42807 for tablet 6423621b78684850a4cb8a6a75a63996, query id is a92f18df4a1647ca81ca4e8bf4c36186
0924 12:46:45.923722 (+ 1533us) tablet_service.cc:2920] Creating iterator
0924 12:46:45.923784 (+ 62us) tablet_service.cc:3297] Waiting safe time to advance
0924 12:46:46.497113 (+573329us) tablet_service.cc:3304] Waiting for operations to commit
0924 12:46:47.352374 (+855261us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1428573 microseconds
0924 12:46:47.352485 (+ 111us) tablet_service.cc:2945] Iterator created
0924 12:46:47.353221 (+ 736us) tablet_service.cc:2967] Iterator init: OK
0924 12:46:47.353258 (+ 37us) tablet_service.cc:3010] has_more: true
0924 12:46:47.353299 (+ 41us) tablet_service.cc:3027] Continuing scan request
0924 12:46:47.353397 (+ 98us) tablet_service.cc:3091] Found scanner 086ad9685825406892f2027b34b42807 for tablet 6423621b78684850a4cb8a6a75a63996, query id is a92f18df4a1647ca81ca4e8bf4c36186
0924 12:46:47.354307 (+ 910us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":23424}
W20260924 12:46:47.355402 7408 scanner-internal.cc:458] Time spent opening tablet: real 1.434s user 0.001s sys 0.000s
W20260924 12:46:47.355618 7407 scanner-internal.cc:458] Time spent opening tablet: real 1.434s user 0.000s sys 0.001s
I20260924 12:46:47.430488 7579 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:47.430720 7579 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:47.431537 7579 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:47.432075 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 39 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:47.432072 7051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 39 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:47.432806 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:47.483664 7579 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:47.485287 7585 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48094 (ReqId={client: b34b87926d454675a830c7cf36b10cff, seq_no=37, attempt_no=0}) took 1714 ms. Trace:
I20260924 12:46:47.485394 7585 rpcz_store.cc:276] 0924 12:46:45.770757 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:45.770818 (+ 61us) service_pool.cc:227] Handling call
0924 12:46:47.485230 (+1714412us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:46:46.497354 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:46.497503 (+ 149us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:46.497509 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:46:46.497512 (+ 3us) tablet.cc:660] Decoding operations
0924 12:46:46.497624 (+ 112us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:46.497636 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:46.497641 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:46.497699 (+ 58us) tablet.cc:699] Row locks acquired
0924 12:46:46.497702 (+ 3us) write_op.cc:260] PREPARE: finished
0924 12:46:46.497748 (+ 46us) write_op.cc:270] Start()
0924 12:46:47.322910 (+825162us) write_op.cc:276] Timestamp: P: 1790254006497705 usec, L: 0
0924 12:46:47.322916 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:46:47.323229 (+ 313us) log.cc:842] Serialized 354 byte log entry
0924 12:46:47.484644 (+161415us) op_driver.cc:444] REPLICATION: finished
0924 12:46:47.484785 (+ 141us) write_op.cc:301] APPLY: starting
0924 12:46:47.484802 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0924 12:46:47.484839 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0924 12:46:47.484871 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:46:47.484933 (+ 62us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:46:47.484960 (+ 27us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:46:47.484964 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:46:47.485036 (+ 72us) log.cc:842] Serialized 22 byte log entry
0924 12:46:47.485107 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:46:47.485122 (+ 15us) write_op.cc:454] Released schema lock
0924 12:46:47.485128 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":53,"num_ops":1,"prepare.queue_time_us":726116,"prepare.run_cpu_time_us":889,"prepare.run_wall_time_us":825956,"replication_time_us":161699}]]}
I20260924 12:46:47.716020 7561 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:47.716248 7561 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:47.716965 7561 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:47.717453 7183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:47.717588 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:47.718168 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9db3b3e723eb41299037b5eab1b00fde; no voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e
I20260924 12:46:47.829316 7579 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:47.829566 7579 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:47.830406 7579 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:47.830962 7051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:47.830967 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:47.832082 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:47.832975 7579 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:48.116735 7561 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:48.354645 7579 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:48.354873 7579 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:48.355582 7579 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:48.356178 7051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 42 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:48.356228 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 42 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:48.356830 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:48.604991 7561 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:48.605304 7561 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:48.606104 7561 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:48.606599 7183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 43 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:48.606774 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 43 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:48.609382 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9db3b3e723eb41299037b5eab1b00fde; no voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e
W20260924 12:46:48.609830 7407 scanner-internal.cc:458] Time spent opening tablet: real 0.626s user 0.000s sys 0.001s
W20260924 12:46:48.609895 7408 scanner-internal.cc:458] Time spent opening tablet: real 0.626s user 0.001s sys 0.000s
I20260924 12:46:48.609961 7561 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:48.958709 7579 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:49.746392 7579 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:49.746623 7579 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:49.747417 7579 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:49.747924 7051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 44 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:49.748200 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 44 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:49.748768 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:49.749922 7601 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48094 (ReqId={client: b34b87926d454675a830c7cf36b10cff, seq_no=41, attempt_no=0}) took 1391 ms. Trace:
I20260924 12:46:49.750058 7601 rpcz_store.cc:276] 0924 12:46:48.358180 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:48.358232 (+ 52us) service_pool.cc:227] Handling call
0924 12:46:49.749863 (+1391631us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:46:48.358686 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:48.358794 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:48.358799 (+ 5us) write_op.cc:435] Acquired schema lock
0924 12:46:48.358801 (+ 2us) tablet.cc:660] Decoding operations
0924 12:46:48.358887 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:48.358894 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:48.358897 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:48.358944 (+ 47us) tablet.cc:699] Row locks acquired
0924 12:46:48.358945 (+ 1us) write_op.cc:260] PREPARE: finished
0924 12:46:48.358982 (+ 37us) write_op.cc:270] Start()
0924 12:46:49.116076 (+757094us) write_op.cc:276] Timestamp: P: 1790254008358977 usec, L: 0
0924 12:46:49.116080 (+ 4us) op_driver.cc:331] REPLICATION: starting
0924 12:46:49.116397 (+ 317us) log.cc:842] Serialized 354 byte log entry
0924 12:46:49.747977 (+631580us) op_driver.cc:444] REPLICATION: finished
0924 12:46:49.748712 (+ 735us) write_op.cc:301] APPLY: starting
0924 12:46:49.748742 (+ 30us) tablet.cc:1365] starting BulkCheckPresence
0924 12:46:49.748780 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
0924 12:46:49.748825 (+ 45us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:46:49.748917 (+ 92us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:46:49.748951 (+ 34us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:46:49.748957 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:46:49.749620 (+ 663us) log.cc:842] Serialized 22 byte log entry
0924 12:46:49.749736 (+ 116us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:46:49.749756 (+ 20us) write_op.cc:454] Released schema lock
0924 12:46:49.749764 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":676,"num_ops":1,"prepare.queue_time_us":45,"prepare.run_cpu_time_us":780,"prepare.run_wall_time_us":757785,"replication_time_us":631870,"thread_start_us":1088,"threads_started":1}]]}
W20260924 12:46:49.752342 7407 scanner-internal.cc:458] Time spent opening tablet: real 0.992s user 0.000s sys 0.001s
W20260924 12:46:49.752377 7408 scanner-internal.cc:458] Time spent opening tablet: real 0.992s user 0.000s sys 0.000s
I20260924 12:46:49.906071 7597 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:49.906740 7597 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:49.907847 7597 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:49.907994 7183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 44 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:49.908257 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 44 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:49.908780 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9db3b3e723eb41299037b5eab1b00fde; no voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e
I20260924 12:46:50.383206 7597 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:50.385983 7607 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48094 (ReqId={client: b34b87926d454675a830c7cf36b10cff, seq_no=42, attempt_no=0}) took 1775 ms. Trace:
I20260924 12:46:50.386262 7607 rpcz_store.cc:276] 0924 12:46:48.610436 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:48.610524 (+ 88us) service_pool.cc:227] Handling call
0924 12:46:50.385923 (+1775399us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:46:49.116511 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:49.116628 (+ 117us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:49.116634 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:46:49.116636 (+ 2us) tablet.cc:660] Decoding operations
0924 12:46:49.116730 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:49.116742 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:49.116747 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:49.116811 (+ 64us) tablet.cc:699] Row locks acquired
0924 12:46:49.116813 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:46:49.116852 (+ 39us) write_op.cc:270] Start()
0924 12:46:49.346002 (+229150us) write_op.cc:276] Timestamp: P: 1790254009116808 usec, L: 0
0924 12:46:49.346007 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:46:49.346256 (+ 249us) log.cc:842] Serialized 354 byte log entry
0924 12:46:50.384941 (+1038685us) op_driver.cc:444] REPLICATION: finished
0924 12:46:50.385438 (+ 497us) write_op.cc:301] APPLY: starting
0924 12:46:50.385457 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0924 12:46:50.385492 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0924 12:46:50.385536 (+ 44us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:46:50.385606 (+ 70us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:46:50.385635 (+ 29us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:46:50.385639 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:46:50.385715 (+ 76us) log.cc:842] Serialized 22 byte log entry
0924 12:46:50.385802 (+ 87us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:46:50.385820 (+ 18us) write_op.cc:454] Released schema lock
0924 12:46:50.385828 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":437,"num_ops":1,"prepare.queue_time_us":505563,"prepare.run_cpu_time_us":783,"prepare.run_wall_time_us":229844,"replication_time_us":1038908,"thread_start_us":328,"threads_started":1}]]}
I20260924 12:46:50.666492 7579 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:50.841475 7579 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:50.841660 7579 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:50.842391 7579 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:50.842981 7051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:50.842993 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:50.843293 7607 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48094 (ReqId={client: b34b87926d454675a830c7cf36b10cff, seq_no=43, attempt_no=0}) took 1090 ms. Trace:
I20260924 12:46:50.843407 7607 rpcz_store.cc:276] 0924 12:46:49.752300 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:49.752383 (+ 83us) service_pool.cc:227] Handling call
0924 12:46:50.843224 (+1090841us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:46:49.752937 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:49.753028 (+ 91us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:49.753033 (+ 5us) write_op.cc:435] Acquired schema lock
0924 12:46:49.753035 (+ 2us) tablet.cc:660] Decoding operations
0924 12:46:49.753106 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:49.753116 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:49.753119 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:49.753159 (+ 40us) tablet.cc:699] Row locks acquired
0924 12:46:49.753161 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:46:49.753193 (+ 32us) write_op.cc:270] Start()
0924 12:46:50.462375 (+709182us) write_op.cc:276] Timestamp: P: 1790254009753140 usec, L: 0
0924 12:46:50.462381 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:46:50.462674 (+ 293us) log.cc:842] Serialized 354 byte log entry
0924 12:46:50.842575 (+379901us) op_driver.cc:444] REPLICATION: finished
0924 12:46:50.842691 (+ 116us) write_op.cc:301] APPLY: starting
0924 12:46:50.842711 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0924 12:46:50.842755 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0924 12:46:50.842780 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:46:50.842865 (+ 85us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:46:50.842900 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:46:50.842905 (+ 5us) write_op.cc:312] APPLY: finished
0924 12:46:50.842998 (+ 93us) log.cc:842] Serialized 22 byte log entry
0924 12:46:50.843102 (+ 104us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:46:50.843121 (+ 19us) write_op.cc:454] Released schema lock
0924 12:46:50.843128 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":32,"num_ops":1,"prepare.queue_time_us":55,"prepare.run_cpu_time_us":732,"prepare.run_wall_time_us":709807,"replication_time_us":380164}]]}
I20260924 12:46:50.844693 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:50.845181 7579 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 12:46:50.845803 7408 scanner-internal.cc:458] Time spent opening tablet: real 0.943s user 0.001s sys 0.000s
W20260924 12:46:50.846752 7407 scanner-internal.cc:458] Time spent opening tablet: real 0.944s user 0.000s sys 0.001s
I20260924 12:46:51.352227 7579 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:51.352483 7579 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:51.353282 7579 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:51.353906 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 47 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:51.353892 7051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 47 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:51.354779 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:51.355064 7579 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:51.403925 7597 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:51.404158 7597 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:51.404943 7597 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:51.405356 7182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:51.405418 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:51.406023 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9db3b3e723eb41299037b5eab1b00fde; no voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e
I20260924 12:46:52.404803 7597 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:52.835870 7616 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:52.836086 7616 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:52.836860 7616 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:52.837381 7051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:52.837380 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:52.838037 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:52.838342 7616 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:52.839468 7623 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48094 (ReqId={client: b34b87926d454675a830c7cf36b10cff, seq_no=45, attempt_no=0}) took 1993 ms. Trace:
I20260924 12:46:52.840000 7623 rpcz_store.cc:276] 0924 12:46:50.845731 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:50.845986 (+ 255us) service_pool.cc:227] Handling call
0924 12:46:52.839405 (+1993419us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:46:50.950719 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:50.950843 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:50.950850 (+ 7us) write_op.cc:435] Acquired schema lock
0924 12:46:50.950852 (+ 2us) tablet.cc:660] Decoding operations
0924 12:46:50.950952 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:50.950964 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:50.950968 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:50.951014 (+ 46us) tablet.cc:699] Row locks acquired
0924 12:46:50.951015 (+ 1us) write_op.cc:260] PREPARE: finished
0924 12:46:50.951054 (+ 39us) write_op.cc:270] Start()
0924 12:46:51.883205 (+932151us) write_op.cc:276] Timestamp: P: 1790254010951025 usec, L: 0
0924 12:46:51.883210 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:46:51.883492 (+ 282us) log.cc:842] Serialized 354 byte log entry
0924 12:46:52.837591 (+954099us) op_driver.cc:444] REPLICATION: finished
0924 12:46:52.838238 (+ 647us) write_op.cc:301] APPLY: starting
0924 12:46:52.838257 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0924 12:46:52.838299 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0924 12:46:52.838329 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:46:52.838408 (+ 79us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:46:52.838444 (+ 36us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:46:52.838450 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:46:52.838556 (+ 106us) log.cc:842] Serialized 22 byte log entry
0924 12:46:52.839268 (+ 712us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:46:52.839288 (+ 20us) write_op.cc:454] Released schema lock
0924 12:46:52.839296 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":582,"num_ops":1,"prepare.queue_time_us":104291,"prepare.run_cpu_time_us":781,"prepare.run_wall_time_us":932850,"replication_time_us":954352,"thread_start_us":472,"threads_started":1}]]}
I20260924 12:46:52.841034 7272 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:48094 (request call id 105) took 1842 ms. Trace:
I20260924 12:46:52.841056 7271 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:48094 (request call id 104) took 1843 ms. Trace:
I20260924 12:46:52.841212 7271 rpcz_store.cc:276] 0924 12:46:50.997363 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:50.997429 (+ 66us) service_pool.cc:227] Handling call
0924 12:46:50.997700 (+ 271us) tablet_service.cc:2780] Created scanner 6f02fce66eb34c9ead94145562e995b3 for tablet 6423621b78684850a4cb8a6a75a63996, query id is e8a71cb5705a4261b186966ec9bde3bd
0924 12:46:50.998432 (+ 732us) tablet_service.cc:2920] Creating iterator
0924 12:46:50.998496 (+ 64us) tablet_service.cc:3297] Waiting safe time to advance
0924 12:46:51.883336 (+884840us) tablet_service.cc:3304] Waiting for operations to commit
0924 12:46:52.838716 (+955380us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1840204 microseconds
0924 12:46:52.838825 (+ 109us) tablet_service.cc:2945] Iterator created
0924 12:46:52.839390 (+ 565us) tablet_service.cc:2967] Iterator init: OK
0924 12:46:52.839427 (+ 37us) tablet_service.cc:3010] has_more: true
0924 12:46:52.839482 (+ 55us) tablet_service.cc:3027] Continuing scan request
0924 12:46:52.839539 (+ 57us) tablet_service.cc:3091] Found scanner 6f02fce66eb34c9ead94145562e995b3 for tablet 6423621b78684850a4cb8a6a75a63996, query id is e8a71cb5705a4261b186966ec9bde3bd
0924 12:46:52.841007 (+ 1468us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":33024}
I20260924 12:46:52.841171 7272 rpcz_store.cc:276] 0924 12:46:50.998056 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:50.998127 (+ 71us) service_pool.cc:227] Handling call
0924 12:46:50.998305 (+ 178us) tablet_service.cc:2780] Created scanner 9c5678c256414d9597c2c7ae312bb8ee for tablet 6423621b78684850a4cb8a6a75a63996, query id is 953d77b15f604b9fb2fd29baa0f798d0
0924 12:46:50.999033 (+ 728us) tablet_service.cc:2920] Creating iterator
0924 12:46:50.999063 (+ 30us) tablet_service.cc:3297] Waiting safe time to advance
0924 12:46:51.883391 (+884328us) tablet_service.cc:3304] Waiting for operations to commit
0924 12:46:52.838715 (+955324us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1839637 microseconds
0924 12:46:52.838825 (+ 110us) tablet_service.cc:2945] Iterator created
0924 12:46:52.839390 (+ 565us) tablet_service.cc:2967] Iterator init: OK
0924 12:46:52.839433 (+ 43us) tablet_service.cc:3010] has_more: true
0924 12:46:52.839466 (+ 33us) tablet_service.cc:3027] Continuing scan request
0924 12:46:52.839536 (+ 70us) tablet_service.cc:3091] Found scanner 9c5678c256414d9597c2c7ae312bb8ee for tablet 6423621b78684850a4cb8a6a75a63996, query id is 953d77b15f604b9fb2fd29baa0f798d0
0924 12:46:52.840977 (+ 1441us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":23168}
W20260924 12:46:52.842005 7408 scanner-internal.cc:458] Time spent opening tablet: real 1.845s user 0.001s sys 0.000s
W20260924 12:46:52.842001 7407 scanner-internal.cc:458] Time spent opening tablet: real 1.845s user 0.000s sys 0.001s
I20260924 12:46:53.288633 7597 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:53.288827 7597 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:53.289533 7597 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:53.290105 7183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:53.290112 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:53.290393 7623 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48094 (ReqId={client: b34b87926d454675a830c7cf36b10cff, seq_no=46, attempt_no=0}) took 1934 ms. Trace:
I20260924 12:46:53.290573 7623 rpcz_store.cc:276] 0924 12:46:51.355585 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:51.355652 (+ 67us) service_pool.cc:227] Handling call
0924 12:46:53.290310 (+1934658us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:46:51.883610 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:51.883734 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:51.883740 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:46:51.883742 (+ 2us) tablet.cc:660] Decoding operations
0924 12:46:51.883862 (+ 120us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:51.883874 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:51.883879 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:51.883935 (+ 56us) tablet.cc:699] Row locks acquired
0924 12:46:51.883937 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:46:51.883982 (+ 45us) write_op.cc:270] Start()
0924 12:46:52.385140 (+501158us) write_op.cc:276] Timestamp: P: 1790254011883925 usec, L: 0
0924 12:46:52.385144 (+ 4us) op_driver.cc:331] REPLICATION: starting
0924 12:46:52.385385 (+ 241us) log.cc:842] Serialized 354 byte log entry
0924 12:46:53.289614 (+904229us) op_driver.cc:444] REPLICATION: finished
0924 12:46:53.289730 (+ 116us) write_op.cc:301] APPLY: starting
0924 12:46:53.289753 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0924 12:46:53.289795 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0924 12:46:53.289838 (+ 43us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:46:53.289920 (+ 82us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:46:53.289955 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:46:53.289961 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:46:53.290047 (+ 86us) log.cc:842] Serialized 22 byte log entry
0924 12:46:53.290138 (+ 91us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:46:53.290160 (+ 22us) write_op.cc:454] Released schema lock
0924 12:46:53.290168 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":61,"num_ops":1,"prepare.queue_time_us":527569,"prepare.run_cpu_time_us":777,"prepare.run_wall_time_us":501844,"replication_time_us":904440}]]}
I20260924 12:46:53.291023 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9db3b3e723eb41299037b5eab1b00fde; no voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e
I20260924 12:46:53.635159 7616 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:53.635412 7616 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:53.636070 7616 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:53.636646 7051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:53.636945 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:53.637460 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:53.719616 7616 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 12:46:53.723483 7407 scanner-internal.cc:458] Time spent opening tablet: real 0.731s user 0.000s sys 0.001s
W20260924 12:46:53.724207 7408 scanner-internal.cc:458] Time spent opening tablet: real 0.731s user 0.001s sys 0.000s
I20260924 12:46:53.795642 7597 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:54.247088 7616 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:54.247327 7616 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:54.248112 7616 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:54.248818 7051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 51 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:54.249068 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 51 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:54.251786 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:54.427227 7616 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:54.652962 7597 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:54.653159 7597 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:54.653861 7597 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:54.654259 7182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 51 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:54.654327 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 51 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:54.655021 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9db3b3e723eb41299037b5eab1b00fde; no voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e
I20260924 12:46:54.899089 7616 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:54.899288 7616 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:54.900012 7616 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:54.900635 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 52 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:54.901191 7051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 52 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:54.901837 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:54.902487 7616 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:55.216645 7616 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:55.216867 7616 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:55.217587 7616 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:55.218096 7051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 53 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:55.218202 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 53 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:55.219086 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:55.221247 7616 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 12:46:55.221594 7408 scanner-internal.cc:458] Time spent opening tablet: real 0.819s user 0.000s sys 0.000s
W20260924 12:46:55.223639 7407 scanner-internal.cc:458] Time spent opening tablet: real 0.821s user 0.000s sys 0.000s
I20260924 12:46:55.407447 7616 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:55.407663 7616 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:55.408363 7616 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:55.409117 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 53 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:55.409360 7051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 53 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:55.410058 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:55.410351 7616 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:55.423766 7616 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:55.423967 7616 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:55.424654 7616 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:55.425074 7051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 53 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:55.425187 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 53 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:55.425882 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:55.426102 7616 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:55.587576 7597 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:55.696674 7597 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:55.696862 7597 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:55.697815 7597 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:55.698211 7182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 53 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:55.698202 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 53 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:55.699028 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9db3b3e723eb41299037b5eab1b00fde; no voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e
I20260924 12:46:55.699271 7597 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:56.452569 7616 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:56.452778 7616 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:56.453653 7616 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:56.454357 7051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 54 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:56.454329 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 54 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:56.455175 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:56.455796 7616 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:56.455840 7651 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48094 (ReqId={client: b34b87926d454675a830c7cf36b10cff, seq_no=51, attempt_no=0}) took 2025 ms. Trace:
I20260924 12:46:56.456014 7651 rpcz_store.cc:276] 0924 12:46:54.430393 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:54.430492 (+ 99us) service_pool.cc:227] Handling call
0924 12:46:56.455822 (+2025330us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:46:54.954300 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6423621b78684850a4cb8a6a75a63996
0924 12:46:54.954448 (+ 148us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:46:54.954480 (+ 32us) write_op.cc:435] Acquired schema lock
0924 12:46:54.954483 (+ 3us) tablet.cc:660] Decoding operations
0924 12:46:54.954583 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:46:54.954595 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:46:54.954600 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:46:54.954657 (+ 57us) tablet.cc:699] Row locks acquired
0924 12:46:54.954659 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:46:54.954702 (+ 43us) write_op.cc:270] Start()
0924 12:46:55.776854 (+822152us) write_op.cc:276] Timestamp: P: 1790254014954651 usec, L: 0
0924 12:46:55.776867 (+ 13us) op_driver.cc:331] REPLICATION: starting
0924 12:46:55.777246 (+ 379us) log.cc:842] Serialized 354 byte log entry
0924 12:46:56.454520 (+677274us) op_driver.cc:444] REPLICATION: finished
0924 12:46:56.455074 (+ 554us) write_op.cc:301] APPLY: starting
0924 12:46:56.455093 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0924 12:46:56.455145 (+ 52us) tablet.cc:1368] finished BulkCheckPresence
0924 12:46:56.455171 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:46:56.455258 (+ 87us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:46:56.455294 (+ 36us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:46:56.455300 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:46:56.455390 (+ 90us) log.cc:842] Serialized 22 byte log entry
0924 12:46:56.455696 (+ 306us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:46:56.455720 (+ 24us) write_op.cc:454] Released schema lock
0924 12:46:56.455728 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":486,"num_ops":1,"prepare.queue_time_us":523284,"prepare.run_cpu_time_us":887,"prepare.run_wall_time_us":823022,"replication_time_us":677623,"spinlock_wait_cycles":316800,"thread_start_us":427,"threads_started":1}]]}
I20260924 12:46:56.562072 7163 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260924 12:46:56.574046 7293 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260924 12:46:56.637270 7597 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:56.637447 7597 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:56.638111 7597 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:56.638806 7183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 54 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:56.639014 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 54 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:56.639504 6987 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9db3b3e723eb41299037b5eab1b00fde; no voters: 6fb4103951824ad892bfc6e460c4069d, 7632672814be4a688ec7ab18a503a58e
I20260924 12:46:56.641620 7032 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260924 12:46:56.651046 7597 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
d0910ced495e438f8b398033c51ef6f5 | 127.5.77.254:46369 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.77.212:45959 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
6fb4103951824ad892bfc6e460c4069d | 127.5.77.194:35473 | HEALTHY | <none> | 0 | 0
7632672814be4a688ec7ab18a503a58e | 127.5.77.195:44947 | HEALTHY | <none> | 1 | 0
9db3b3e723eb41299037b5eab1b00fde | 127.5.77.193:36435 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
inject_latency_ms_before_starting_op | 1000 | hidden,advanced | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.5.77.193 | experimental | 127.5.77.193:36435
local_ip_for_outbound_sockets | 127.5.77.194 | experimental | 127.5.77.194:35473
local_ip_for_outbound_sockets | 127.5.77.195 | experimental | 127.5.77.195:44947
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.5.77.193:36435
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.77.194:35473
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.77.195:44947
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.77.212:45959 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260924 12:46:56.690160 5431 log_verifier.cc:126] Checking tablet 6423621b78684850a4cb8a6a75a63996
I20260924 12:46:56.759709 5431 log_verifier.cc:177] Verified matching terms for 54 ops in tablet 6423621b78684850a4cb8a6a75a63996
I20260924 12:46:56.789143 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 6970
I20260924 12:46:56.798005 7092 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:56.876291 7616 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:56.876480 7616 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:56.877871 7616 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:56.878831 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 54 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
W20260924 12:46:56.885640 7118 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.5.77.193:36435 timed out after 0.008s (SENT) [suppressed 1 similar messages]
W20260924 12:46:56.911605 7118 leader_election.cc:334] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435): Timed out: RequestConsensusVote RPC to 127.5.77.193:36435 timed out after 0.008s (SENT)
I20260924 12:46:56.911780 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:56.912035 7616 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:56.956842 7616 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:56.957013 7616 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:56.957607 7616 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:56.959594 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 54 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
W20260924 12:46:56.973155 7118 leader_election.cc:334] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435): Timed out: RequestConsensusVote RPC to 127.5.77.193:36435 timed out after 0.007s (SENT)
I20260924 12:46:56.973315 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:56.973567 7616 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:56.987190 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 54 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:56.987637 7051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 54 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:56.987766 7050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 54 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:56.987977 7597 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:56.988147 7597 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:56.988651 7049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 54 } } ignore_live_leader: false dest_uuid: "9db3b3e723eb41299037b5eab1b00fde" is_pre_election: true
I20260924 12:46:56.988793 7597 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:56.989288 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 6970
I20260924 12:46:56.989392 7052 raft_consensus.cc:2454] T 6423621b78684850a4cb8a6a75a63996 P 9db3b3e723eb41299037b5eab1b00fde [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fb4103951824ad892bfc6e460c4069d in term 3.
I20260924 12:46:56.990144 7183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 54 } } ignore_live_leader: false dest_uuid: "6fb4103951824ad892bfc6e460c4069d" is_pre_election: true
I20260924 12:46:56.990877 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "9db3b3e723eb41299037b5eab1b00fde" candidate_term: 4 candidate_status { last_received { term: 3 index: 54 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
I20260924 12:46:56.994679 7616 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:56.994863 7616 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:56.995345 7616 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:56.995844 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 54 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
W20260924 12:46:57.012398 7118 leader_election.cc:334] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435): Timed out: RequestConsensusVote RPC to 127.5.77.193:36435 timed out after 0.008s (SENT)
I20260924 12:46:57.012533 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:57.012771 7616 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 12:46:57.017210 7118 connection.cc:535] client connection to 127.5.77.193:36435 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 12:46:57.017248 7117 connection.cc:535] server connection from 127.5.77.193:45955 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 12:46:57.017688 7248 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.77.193:36435 (error 108) [suppressed 1 similar messages]
I20260924 12:46:57.017925 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 7101
W20260924 12:46:57.018481 7247 connection.cc:535] server connection from 127.5.77.193:37605 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 12:46:57.023955 7248 consensus_peers.cc:487] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e -> Peer 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435): Couldn't send request to peer 9db3b3e723eb41299037b5eab1b00fde. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:36435: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 12:46:57.025275 7616 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:57.025451 7616 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:57.026065 7616 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:57.028142 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 54 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
W20260924 12:46:57.029096 7118 leader_election.cc:334] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435): Network error: Client connection negotiation failed: client connection to 127.5.77.193:36435: connect: Connection refused (error 111)
I20260924 12:46:57.029232 7118 leader_election.cc:302] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb4103951824ad892bfc6e460c4069d; no voters: 7632672814be4a688ec7ab18a503a58e, 9db3b3e723eb41299037b5eab1b00fde
I20260924 12:46:57.029524 7616 raft_consensus.cc:2726] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:46:57.030342 7223 generic_service.cc:196] Checking for leaks (request via RPC)
W20260924 12:46:57.036367 7248 consensus_peers.cc:487] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e -> Peer 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435): Couldn't send request to peer 9db3b3e723eb41299037b5eab1b00fde. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:36435: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260924 12:46:57.048640 7248 consensus_peers.cc:487] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e -> Peer 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435): Couldn't send request to peer 9db3b3e723eb41299037b5eab1b00fde. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:36435: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260924 12:46:57.059783 7248 consensus_peers.cc:487] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e -> Peer 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435): Couldn't send request to peer 9db3b3e723eb41299037b5eab1b00fde. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:36435: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260924 12:46:57.071020 7248 consensus_peers.cc:487] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e -> Peer 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435): Couldn't send request to peer 9db3b3e723eb41299037b5eab1b00fde. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:36435: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260924 12:46:57.082149 7248 consensus_peers.cc:487] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e -> Peer 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435): Couldn't send request to peer 9db3b3e723eb41299037b5eab1b00fde. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:36435: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260924 12:46:57.093498 7248 consensus_peers.cc:487] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e -> Peer 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435): Couldn't send request to peer 9db3b3e723eb41299037b5eab1b00fde. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:36435: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260924 12:46:57.104633 7248 consensus_peers.cc:487] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e -> Peer 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435): Couldn't send request to peer 9db3b3e723eb41299037b5eab1b00fde. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:36435: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260924 12:46:57.115959 7248 consensus_peers.cc:487] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e -> Peer 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435): Couldn't send request to peer 9db3b3e723eb41299037b5eab1b00fde. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:36435: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260924 12:46:57.127187 7248 consensus_peers.cc:487] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e -> Peer 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435): Couldn't send request to peer 9db3b3e723eb41299037b5eab1b00fde. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:36435: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260924 12:46:57.138778 7248 consensus_peers.cc:487] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e -> Peer 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435): Couldn't send request to peer 9db3b3e723eb41299037b5eab1b00fde. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:36435: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260924 12:46:57.150117 7248 consensus_peers.cc:487] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e -> Peer 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435): Couldn't send request to peer 9db3b3e723eb41299037b5eab1b00fde. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:36435: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260924 12:46:57.161079 7248 consensus_peers.cc:487] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e -> Peer 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435): Couldn't send request to peer 9db3b3e723eb41299037b5eab1b00fde. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:36435: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260924 12:46:57.172307 7248 consensus_peers.cc:487] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e -> Peer 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435): Couldn't send request to peer 9db3b3e723eb41299037b5eab1b00fde. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:36435: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260924 12:46:57.183676 7248 consensus_peers.cc:487] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e -> Peer 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435): Couldn't send request to peer 9db3b3e723eb41299037b5eab1b00fde. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:36435: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20260924 12:46:57.194773 7248 consensus_peers.cc:487] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e -> Peer 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435): Couldn't send request to peer 9db3b3e723eb41299037b5eab1b00fde. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:36435: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20260924 12:46:57.206044 7248 consensus_peers.cc:487] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e -> Peer 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435): Couldn't send request to peer 9db3b3e723eb41299037b5eab1b00fde. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:36435: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20260924 12:46:57.213593 7616 raft_consensus.cc:482] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7632672814be4a688ec7ab18a503a58e)
I20260924 12:46:57.213790 7616 raft_consensus.cc:504] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb4103951824ad892bfc6e460c4069d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35473 } } peers { permanent_uuid: "9db3b3e723eb41299037b5eab1b00fde" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 36435 } } peers { permanent_uuid: "7632672814be4a688ec7ab18a503a58e" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 44947 } }
I20260924 12:46:57.214372 7616 leader_election.cc:288] T 6423621b78684850a4cb8a6a75a63996 P 6fb4103951824ad892bfc6e460c4069d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435), 7632672814be4a688ec7ab18a503a58e (127.5.77.195:44947)
I20260924 12:46:57.214910 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 7101
I20260924 12:46:57.216146 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6423621b78684850a4cb8a6a75a63996" candidate_uuid: "6fb4103951824ad892bfc6e460c4069d" candidate_term: 4 candidate_status { last_received { term: 3 index: 54 } } ignore_live_leader: false dest_uuid: "7632672814be4a688ec7ab18a503a58e" is_pre_election: true
W20260924 12:46:57.220311 7248 consensus_peers.cc:487] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e -> Peer 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435): Couldn't send request to peer 9db3b3e723eb41299037b5eab1b00fde. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:36435: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20260924 12:46:57.231542 7248 consensus_peers.cc:487] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e -> Peer 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435): Couldn't send request to peer 9db3b3e723eb41299037b5eab1b00fde. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:36435: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20260924 12:46:57.239859 7249 connection.cc:535] server connection from 127.5.77.194:59699 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 12:46:57.240227 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 7231
W20260924 12:46:57.241364 7248 consensus_peers.cc:487] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e -> Peer 6fb4103951824ad892bfc6e460c4069d (127.5.77.194:35473): Couldn't send request to peer 6fb4103951824ad892bfc6e460c4069d. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.194:35473: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 12:46:57.244014 7248 consensus_peers.cc:487] T 6423621b78684850a4cb8a6a75a63996 P 7632672814be4a688ec7ab18a503a58e -> Peer 9db3b3e723eb41299037b5eab1b00fde (127.5.77.193:36435): Couldn't send request to peer 9db3b3e723eb41299037b5eab1b00fde. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:36435: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20260924 12:46:57.247296 7353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:57.417425 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 7231
I20260924 12:46:57.441375 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 6880
I20260924 12:46:57.442338 6941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:57.551630 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 6880
2026-09-24T12:46:57Z chronyd exiting
[ OK ] RaftConsensusElectionITest.ChurnyElections (35064 ms)
[ RUN ] RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency
I20260924 12:46:57.589004 5431 test_util.cc:268] Using random seed: -20533515
I20260924 12:46:57.590652 5431 ts_itest-base.cc:112] Starting cluster with:
I20260924 12:46:57.590713 5431 ts_itest-base.cc:113] --------------
I20260924 12:46:57.590765 5431 ts_itest-base.cc:114] 3 tablet servers
I20260924 12:46:57.590811 5431 ts_itest-base.cc:115] 3 replicas per TS
I20260924 12:46:57.590869 5431 ts_itest-base.cc:116] --------------
2026-09-24T12:46:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:46:57Z Disabled control of system clock
I20260924 12:46:57.613350 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.77.254:33661
--webserver_interface=127.5.77.254
--webserver_port=0
--builtin_ntp_servers=127.5.77.212:46531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.77.254:33661 with env {}
W20260924 12:46:57.920194 7705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:57.920506 7705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:57.920629 7705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:57.928376 7705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:46:57.928504 7705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:57.928561 7705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:46:57.928648 7705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:46:57.938320 7705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:46531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.77.254:33661
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.77.254:33661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.77.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:57.939743 7705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:57.941638 7705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:57.953797 7711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:57.953871 7713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:57.954121 7705 server_base.cc:1034] running on GCE node
W20260924 12:46:57.953862 7710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:57.955168 7705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:57.956636 7705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:57.957856 7705 hybrid_clock.cc:648] HybridClock initialized: now 1790254017957779 us; error 61 us; skew 500 ppm
I20260924 12:46:57.958225 7705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:46:57.960516 7705 webserver.cc:460] Webserver started at http://127.5.77.254:46077/ using document root <none> and password file <none>
I20260924 12:46:57.961186 7705 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:57.961302 7705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:57.961587 7705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:57.964303 7705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "cb77ffc814ce46ab999a63f4f6c7fa14"
format_stamp: "Formatted at 2026-09-24 12:46:57 on dist-test-slave-b8q3"
I20260924 12:46:57.965052 7705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "cb77ffc814ce46ab999a63f4f6c7fa14"
format_stamp: "Formatted at 2026-09-24 12:46:57 on dist-test-slave-b8q3"
I20260924 12:46:57.970180 7705 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 12:46:57.973534 7719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:57.975054 7705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:46:57.975207 7705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
uuid: "cb77ffc814ce46ab999a63f4f6c7fa14"
format_stamp: "Formatted at 2026-09-24 12:46:57 on dist-test-slave-b8q3"
I20260924 12:46:57.975394 7705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:57.990242 7705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:57.991039 7705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:57.991318 7705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:58.008080 7705 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.254:33661
I20260924 12:46:58.008101 7770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.254:33661 every 8 connection(s)
I20260924 12:46:58.009436 7705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260924 12:46:58.013207 7771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:58.019544 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 7705
I20260924 12:46:58.019723 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260924 12:46:58.024070 7771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb77ffc814ce46ab999a63f4f6c7fa14: Bootstrap starting.
I20260924 12:46:58.027241 7771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb77ffc814ce46ab999a63f4f6c7fa14: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:58.028259 7771 log.cc:824] T 00000000000000000000000000000000 P cb77ffc814ce46ab999a63f4f6c7fa14: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:58.030872 7771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb77ffc814ce46ab999a63f4f6c7fa14: No bootstrap required, opened a new log
I20260924 12:46:58.039072 7771 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb77ffc814ce46ab999a63f4f6c7fa14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb77ffc814ce46ab999a63f4f6c7fa14" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 33661 } }
I20260924 12:46:58.039434 7771 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb77ffc814ce46ab999a63f4f6c7fa14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:58.039529 7771 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb77ffc814ce46ab999a63f4f6c7fa14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb77ffc814ce46ab999a63f4f6c7fa14, State: Initialized, Role: FOLLOWER
I20260924 12:46:58.040093 7771 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb77ffc814ce46ab999a63f4f6c7fa14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb77ffc814ce46ab999a63f4f6c7fa14" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 33661 } }
I20260924 12:46:58.040295 7771 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb77ffc814ce46ab999a63f4f6c7fa14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:46:58.040409 7771 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb77ffc814ce46ab999a63f4f6c7fa14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:46:58.040560 7771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb77ffc814ce46ab999a63f4f6c7fa14 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:58.042546 7771 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb77ffc814ce46ab999a63f4f6c7fa14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb77ffc814ce46ab999a63f4f6c7fa14" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 33661 } }
I20260924 12:46:58.043071 7771 leader_election.cc:302] T 00000000000000000000000000000000 P cb77ffc814ce46ab999a63f4f6c7fa14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb77ffc814ce46ab999a63f4f6c7fa14; no voters:
I20260924 12:46:58.043614 7771 leader_election.cc:288] T 00000000000000000000000000000000 P cb77ffc814ce46ab999a63f4f6c7fa14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:46:58.043744 7776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb77ffc814ce46ab999a63f4f6c7fa14 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:58.044430 7776 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb77ffc814ce46ab999a63f4f6c7fa14 [term 1 LEADER]: Becoming Leader. State: Replica: cb77ffc814ce46ab999a63f4f6c7fa14, State: Running, Role: LEADER
I20260924 12:46:58.044929 7776 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb77ffc814ce46ab999a63f4f6c7fa14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb77ffc814ce46ab999a63f4f6c7fa14" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 33661 } }
I20260924 12:46:58.045581 7771 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb77ffc814ce46ab999a63f4f6c7fa14 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:46:58.047583 7777 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb77ffc814ce46ab999a63f4f6c7fa14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb77ffc814ce46ab999a63f4f6c7fa14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb77ffc814ce46ab999a63f4f6c7fa14" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 33661 } } }
I20260924 12:46:58.047699 7778 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb77ffc814ce46ab999a63f4f6c7fa14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb77ffc814ce46ab999a63f4f6c7fa14. Latest consensus state: current_term: 1 leader_uuid: "cb77ffc814ce46ab999a63f4f6c7fa14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb77ffc814ce46ab999a63f4f6c7fa14" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 33661 } } }
I20260924 12:46:58.047974 7777 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb77ffc814ce46ab999a63f4f6c7fa14 [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:58.048012 7778 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb77ffc814ce46ab999a63f4f6c7fa14 [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:58.049908 7784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:46:58.055027 7784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:46:58.063386 7784 catalog_manager.cc:1347] Generated new cluster ID: bc8e7efd3dbb4910ab0a074a299d3507
I20260924 12:46:58.063505 7784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:46:58.085911 7784 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:46:58.087339 7784 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:46:58.100164 7784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb77ffc814ce46ab999a63f4f6c7fa14: Generated new TSK 0
I20260924 12:46:58.101020 7784 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:46:58.112238 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.193:0
--local_ip_for_outbound_sockets=127.5.77.193
--webserver_interface=127.5.77.193
--webserver_port=0
--tserver_master_addrs=127.5.77.254:33661
--builtin_ntp_servers=127.5.77.212:46531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000
--consensus_inject_latency_ms_in_notifications=50 with env {}
W20260924 12:46:58.446803 7795 flags.cc:425] Enabled unsafe flag: --consensus_inject_latency_ms_in_notifications=50
W20260924 12:46:58.447080 7795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:58.447142 7795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:58.447214 7795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:58.454312 7795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:58.454540 7795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.193
I20260924 12:46:58.463379 7795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:46531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_inject_latency_ms_in_notifications=50
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.77.193
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.5.77.254:33661
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.193
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:58.464568 7795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:58.466188 7795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:58.478188 7800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:58.478148 7801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:58.478209 7803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:58.478780 7795 server_base.cc:1034] running on GCE node
I20260924 12:46:58.479362 7795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:58.480273 7795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:58.481509 7795 hybrid_clock.cc:648] HybridClock initialized: now 1790254018481449 us; error 43 us; skew 500 ppm
I20260924 12:46:58.481817 7795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:46:58.483963 7795 webserver.cc:460] Webserver started at http://127.5.77.193:44535/ using document root <none> and password file <none>
I20260924 12:46:58.484615 7795 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:58.484709 7795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:58.484992 7795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:58.489075 7795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9"
format_stamp: "Formatted at 2026-09-24 12:46:58 on dist-test-slave-b8q3"
I20260924 12:46:58.489817 7795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9"
format_stamp: "Formatted at 2026-09-24 12:46:58 on dist-test-slave-b8q3"
I20260924 12:46:58.494726 7795 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 12:46:58.497906 7809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:58.499238 7795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:46:58.499378 7795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9"
format_stamp: "Formatted at 2026-09-24 12:46:58 on dist-test-slave-b8q3"
I20260924 12:46:58.499562 7795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:58.518603 7795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:58.519377 7795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:58.519639 7795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:58.520557 7795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:58.521893 7795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:58.521986 7795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:58.522089 7795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:58.522162 7795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:58.551397 7795 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.193:39579
I20260924 12:46:58.551453 7921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.193:39579 every 8 connection(s)
I20260924 12:46:58.552667 7795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260924 12:46:58.553143 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 7795
I20260924 12:46:58.553328 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260924 12:46:58.556929 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.194:0
--local_ip_for_outbound_sockets=127.5.77.194
--webserver_interface=127.5.77.194
--webserver_port=0
--tserver_master_addrs=127.5.77.254:33661
--builtin_ntp_servers=127.5.77.212:46531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000
--consensus_inject_latency_ms_in_notifications=50 with env {}
I20260924 12:46:58.567267 7922 heartbeater.cc:347] Connected to a master server at 127.5.77.254:33661
I20260924 12:46:58.567544 7922 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:58.568250 7922 heartbeater.cc:511] Master 127.5.77.254:33661 requested a full tablet report, sending...
I20260924 12:46:58.570515 7736 ts_manager.cc:194] Registered new tserver with Master: 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:46:58.572615 7736 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.193:41829
W20260924 12:46:58.866587 7926 flags.cc:425] Enabled unsafe flag: --consensus_inject_latency_ms_in_notifications=50
W20260924 12:46:58.866892 7926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:58.866954 7926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:58.867025 7926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:58.875677 7926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:58.875913 7926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.194
I20260924 12:46:58.885823 7926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:46531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_inject_latency_ms_in_notifications=50
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.77.194
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.5.77.254:33661
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.194
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:58.887174 7926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:58.888952 7926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:58.901513 7934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:58.901611 7931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:58.901654 7932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:58.902822 7926 server_base.cc:1034] running on GCE node
I20260924 12:46:58.903472 7926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:58.904454 7926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:58.905673 7926 hybrid_clock.cc:648] HybridClock initialized: now 1790254018905619 us; error 39 us; skew 500 ppm
I20260924 12:46:58.905992 7926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:46:58.908177 7926 webserver.cc:460] Webserver started at http://127.5.77.194:34271/ using document root <none> and password file <none>
I20260924 12:46:58.908864 7926 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:58.908968 7926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:58.909265 7926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:58.912166 7926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d0a6698cb7734148a8c2a168182e095e"
format_stamp: "Formatted at 2026-09-24 12:46:58 on dist-test-slave-b8q3"
I20260924 12:46:58.912981 7926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d0a6698cb7734148a8c2a168182e095e"
format_stamp: "Formatted at 2026-09-24 12:46:58 on dist-test-slave-b8q3"
I20260924 12:46:58.918380 7926 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 12:46:58.921819 7940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:58.923190 7926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 12:46:58.923350 7926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d0a6698cb7734148a8c2a168182e095e"
format_stamp: "Formatted at 2026-09-24 12:46:58 on dist-test-slave-b8q3"
I20260924 12:46:58.923570 7926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:58.945149 7926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:58.946074 7926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:58.946400 7926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:58.947402 7926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:58.949038 7926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:58.949144 7926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:58.949285 7926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:58.949369 7926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:58.980783 7926 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.194:39283
I20260924 12:46:58.980808 8052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.194:39283 every 8 connection(s)
I20260924 12:46:58.982328 7926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260924 12:46:58.985116 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 7926
I20260924 12:46:58.985265 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260924 12:46:58.988107 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.195:0
--local_ip_for_outbound_sockets=127.5.77.195
--webserver_interface=127.5.77.195
--webserver_port=0
--tserver_master_addrs=127.5.77.254:33661
--builtin_ntp_servers=127.5.77.212:46531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000
--consensus_inject_latency_ms_in_notifications=50 with env {}
I20260924 12:46:58.997026 8053 heartbeater.cc:347] Connected to a master server at 127.5.77.254:33661
I20260924 12:46:58.997354 8053 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:58.998070 8053 heartbeater.cc:511] Master 127.5.77.254:33661 requested a full tablet report, sending...
I20260924 12:46:58.999488 7736 ts_manager.cc:194] Registered new tserver with Master: d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:46:59.000245 7736 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.194:44639
W20260924 12:46:59.301093 8056 flags.cc:425] Enabled unsafe flag: --consensus_inject_latency_ms_in_notifications=50
W20260924 12:46:59.301437 8056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:59.301527 8056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:59.301630 8056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:59.309139 8056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:59.309325 8056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.195
I20260924 12:46:59.318684 8056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:46531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_inject_latency_ms_in_notifications=50
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.77.195
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.5.77.254:33661
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.195
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:59.319972 8056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:59.321722 8056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:59.334703 8061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:59.334666 8064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:59.334723 8062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:59.335448 8056 server_base.cc:1034] running on GCE node
I20260924 12:46:59.336074 8056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:59.336987 8056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:59.338186 8056 hybrid_clock.cc:648] HybridClock initialized: now 1790254019338139 us; error 33 us; skew 500 ppm
I20260924 12:46:59.338536 8056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:46:59.340935 8056 webserver.cc:460] Webserver started at http://127.5.77.195:45227/ using document root <none> and password file <none>
I20260924 12:46:59.341629 8056 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:59.341727 8056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:59.342016 8056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:59.344614 8056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a5592d2da7454940a8bade621094bf73"
format_stamp: "Formatted at 2026-09-24 12:46:59 on dist-test-slave-b8q3"
I20260924 12:46:59.345345 8056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a5592d2da7454940a8bade621094bf73"
format_stamp: "Formatted at 2026-09-24 12:46:59 on dist-test-slave-b8q3"
I20260924 12:46:59.350549 8056 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 12:46:59.353732 8070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:59.355113 8056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:46:59.355283 8056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a5592d2da7454940a8bade621094bf73"
format_stamp: "Formatted at 2026-09-24 12:46:59 on dist-test-slave-b8q3"
I20260924 12:46:59.355494 8056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:59.367874 8056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:59.368657 8056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:59.368901 8056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:59.369779 8056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:59.371204 8056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:59.371297 8056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:59.371397 8056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:59.371469 8056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:59.400817 8056 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.195:33859
I20260924 12:46:59.400887 8182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.195:33859 every 8 connection(s)
I20260924 12:46:59.402115 8056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260924 12:46:59.407166 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 8056
I20260924 12:46:59.407323 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260924 12:46:59.414844 8183 heartbeater.cc:347] Connected to a master server at 127.5.77.254:33661
I20260924 12:46:59.415117 8183 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:59.415781 8183 heartbeater.cc:511] Master 127.5.77.254:33661 requested a full tablet report, sending...
I20260924 12:46:59.417022 7736 ts_manager.cc:194] Registered new tserver with Master: a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859)
I20260924 12:46:59.417783 7736 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.195:50863
I20260924 12:46:59.422281 5431 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:46:59.425449 5431 test_util.cc:268] Using random seed: -18697084
I20260924 12:46:59.444113 7736 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53286:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:46:59.445662 7736 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:46:59.485687 8118 tablet_service.cc:1467] Processing CreateTablet for tablet 0121d1741e224a7ba39ced9c1fdb2b4b (DEFAULT_TABLE table=test-workload [id=f7080e4ffff44a22a72b8608c75dca3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:59.485687 7857 tablet_service.cc:1467] Processing CreateTablet for tablet 0121d1741e224a7ba39ced9c1fdb2b4b (DEFAULT_TABLE table=test-workload [id=f7080e4ffff44a22a72b8608c75dca3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:59.485836 7988 tablet_service.cc:1467] Processing CreateTablet for tablet 0121d1741e224a7ba39ced9c1fdb2b4b (DEFAULT_TABLE table=test-workload [id=f7080e4ffff44a22a72b8608c75dca3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:59.487082 7988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0121d1741e224a7ba39ced9c1fdb2b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:59.487114 8118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0121d1741e224a7ba39ced9c1fdb2b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:59.487114 7857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0121d1741e224a7ba39ced9c1fdb2b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:59.498169 8201 tablet_bootstrap.cc:492] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e: Bootstrap starting.
I20260924 12:46:59.500383 8203 tablet_bootstrap.cc:492] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9: Bootstrap starting.
I20260924 12:46:59.501900 8201 tablet_bootstrap.cc:654] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:59.503751 8202 tablet_bootstrap.cc:492] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73: Bootstrap starting.
I20260924 12:46:59.503319 8201 log.cc:824] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:59.504245 8203 tablet_bootstrap.cc:654] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:59.505535 8203 log.cc:824] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:59.506958 8201 tablet_bootstrap.cc:492] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e: No bootstrap required, opened a new log
I20260924 12:46:59.507236 8201 ts_tablet_manager.cc:1397] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 12:46:59.507642 8202 tablet_bootstrap.cc:654] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:59.508210 8203 tablet_bootstrap.cc:492] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9: No bootstrap required, opened a new log
I20260924 12:46:59.508467 8203 ts_tablet_manager.cc:1397] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 12:46:59.508553 8202 log.cc:824] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:59.510742 8202 tablet_bootstrap.cc:492] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73: No bootstrap required, opened a new log
I20260924 12:46:59.510969 8202 ts_tablet_manager.cc:1397] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260924 12:46:59.517295 8201 raft_consensus.cc:348] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:46:59.517740 8201 raft_consensus.cc:374] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:59.517848 8201 raft_consensus.cc:729] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a6698cb7734148a8c2a168182e095e, State: Initialized, Role: FOLLOWER
I20260924 12:46:59.517831 8202 raft_consensus.cc:348] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:46:59.518203 8202 raft_consensus.cc:374] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:59.518298 8202 raft_consensus.cc:729] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5592d2da7454940a8bade621094bf73, State: Initialized, Role: FOLLOWER
I20260924 12:46:59.518297 8203 raft_consensus.cc:348] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:46:59.518493 8201 consensus_queue.cc:260] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:46:59.518718 8203 raft_consensus.cc:374] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:59.518805 8203 raft_consensus.cc:729] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, State: Initialized, Role: FOLLOWER
I20260924 12:46:59.518854 8202 consensus_queue.cc:260] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:46:59.520337 8201 ts_tablet_manager.cc:1428] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e: Time spent starting tablet: real 0.013s user 0.006s sys 0.007s
I20260924 12:46:59.520852 8053 heartbeater.cc:503] Master 127.5.77.254:33661 was elected leader, sending a full tablet report...
I20260924 12:46:59.526999 8211 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:46:59.527058 8183 heartbeater.cc:503] Master 127.5.77.254:33661 was elected leader, sending a full tablet report...
I20260924 12:46:59.527334 8211 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:46:59.528375 8210 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:46:59.528635 8210 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:46:59.532260 8210 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:46:59.533711 8211 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:46:59.535764 8202 ts_tablet_manager.cc:1428] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73: Time spent starting tablet: real 0.025s user 0.018s sys 0.005s
W20260924 12:46:59.540032 8073 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.5.77.193:39579 for RPC RequestConsensusVote timed out after 0.006s (ON_OUTBOUND_QUEUE)
I20260924 12:46:59.544399 8203 consensus_queue.cc:260] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:46:59.545066 8208 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:46:59.545367 8208 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
W20260924 12:46:59.547487 7944 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.5.77.195:33859 for RPC RequestConsensusVote timed out after 0.008s (ON_OUTBOUND_QUEUE)
I20260924 12:46:59.549861 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:46:59.550372 8008 raft_consensus.cc:2454] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5592d2da7454940a8bade621094bf73 in term 0.
W20260924 12:46:59.551952 8073 leader_election.cc:334] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579): Timed out: connection negotiation to 127.5.77.193:39579 for RPC RequestConsensusVote timed out after 0.006s (ON_OUTBOUND_QUEUE)
W20260924 12:46:59.556792 7944 leader_election.cc:334] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859): Timed out: connection negotiation to 127.5.77.195:33859 for RPC RequestConsensusVote timed out after 0.008s (ON_OUTBOUND_QUEUE)
I20260924 12:46:59.558516 7922 heartbeater.cc:503] Master 127.5.77.254:33661 was elected leader, sending a full tablet report...
W20260924 12:46:59.560158 8074 leader_election.cc:334] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283): Timed out: RequestConsensusVote RPC to 127.5.77.194:39283 timed out after 0.006s (SENT)
I20260924 12:46:59.560290 8074 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:46:59.560811 8209 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 12:46:59.564215 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "d0a6698cb7734148a8c2a168182e095e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:46:59.564828 7877 raft_consensus.cc:2454] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a6698cb7734148a8c2a168182e095e in term 0.
I20260924 12:46:59.565487 7943 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e; no voters: a5592d2da7454940a8bade621094bf73
I20260924 12:46:59.565982 8210 raft_consensus.cc:2781] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:46:59.566138 8210 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:46:59.566248 8210 raft_consensus.cc:3037] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:59.568596 8209 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:46:59.568792 8209 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:46:59.569329 8210 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:46:59.569988 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:46:59.570277 8008 raft_consensus.cc:2379] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5592d2da7454940a8bade621094bf73 in current term 1: Already voted for candidate d0a6698cb7734148a8c2a168182e095e in this term.
I20260924 12:46:59.570442 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "d0a6698cb7734148a8c2a168182e095e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9"
I20260924 12:46:59.570652 7877 raft_consensus.cc:3037] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:59.571482 8138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "d0a6698cb7734148a8c2a168182e095e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73"
I20260924 12:46:59.571810 8138 raft_consensus.cc:3037] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:59.548203 8208 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:46:59.573537 7877 raft_consensus.cc:2454] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a6698cb7734148a8c2a168182e095e in term 1.
I20260924 12:46:59.570009 8210 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [CANDIDATE]: Term 1 election: Requested vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:46:59.574129 7943 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e; no voters:
I20260924 12:46:59.574488 8210 raft_consensus.cc:2781] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:59.574652 8203 ts_tablet_manager.cc:1428] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9: Time spent starting tablet: real 0.066s user 0.019s sys 0.010s
I20260924 12:46:59.574611 8138 raft_consensus.cc:2454] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a6698cb7734148a8c2a168182e095e in term 1.
I20260924 12:46:59.574935 8210 raft_consensus.cc:686] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [term 1 LEADER]: Becoming Leader. State: Replica: d0a6698cb7734148a8c2a168182e095e, State: Running, Role: LEADER
I20260924 12:46:59.575269 8209 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:46:59.575845 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:46:59.576167 7877 raft_consensus.cc:2379] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5592d2da7454940a8bade621094bf73 in current term 1: Already voted for candidate d0a6698cb7734148a8c2a168182e095e in this term.
W20260924 12:46:59.576315 7813 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.5.77.195:33859 timed out after 0.006s (ON_OUTBOUND_QUEUE)
I20260924 12:46:59.575449 8210 consensus_queue.cc:237] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:46:59.576895 8073 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:46:59.577167 8209 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 12:46:59.581648 7735 catalog_manager.cc:5495] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e reported cstate change: term changed from 0 to 1, leader changed from <none> to d0a6698cb7734148a8c2a168182e095e (127.5.77.194). New cstate: current_term: 1 leader_uuid: "d0a6698cb7734148a8c2a168182e095e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } health_report { overall_health: UNKNOWN } } }
I20260924 12:46:59.583623 8137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:46:59.584038 8137 raft_consensus.cc:2379] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17f2a7fd399b4c8b9b96a1d8ec5d39e9 in current term 1: Already voted for candidate d0a6698cb7734148a8c2a168182e095e in this term.
I20260924 12:46:59.584481 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:46:59.584511 8210 consensus_queue.cc:1035] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:59.585203 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
W20260924 12:46:59.585359 8210 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260924 12:46:59.585438 8210 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260924 12:46:59.585565 8208 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 12:46:59.734298 8054 tablet.cc:2369] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 12:46:59.771803 7837 tablet_service.cc:2970] error setting up scanner 1a85a76437714b1f9bfe57a0fefc2a1c with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.143s (maximum allowed 0.100s): new_scan_request { tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\357\271\324\325\006\"3\n\005slave\022*\n f7080e4ffff44a22a72b8608c75dca3e\020\001\030\001 \001(\001" signature: "<redacted>""\225\227\301p\373\026\334\232\225\005f\212x\357\265\2618\021`0\217\300\347\n\235Q\\t\253cU\353d\3312a\337\313\342\335X\221\266\t\221\034\225jV\301,:\201\203\335\022)\235}P\271\374\270\330" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "dcfcbbc642ea47f3b43d5c78efc7a745"
W20260924 12:46:59.771827 7836 tablet_service.cc:2970] error setting up scanner 5bdeb71669964908808c930f454b5ee5 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.143s (maximum allowed 0.100s): new_scan_request { tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\357\271\324\325\006\"3\n\005slave\022*\n f7080e4ffff44a22a72b8608c75dca3e\020\001\030\001 \001(\001" signature: "<redacted>""\225\227\301p\373\026\334\232\225\005f\212x\357\265\2618\021`0\217\300\347\n\235Q\\t\253cU\353d\3312a\337\313\342\335X\221\266\t\221\034\225jV\301,:\201\203\335\022)\235}P\271\374\270\330" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b1e701df8a3f4c2b8c2f0ef68abb2436"
I20260924 12:46:59.774749 8208 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:46:59.774917 8208 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:46:59.775419 8208 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:46:59.775823 8137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:46:59.775861 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:46:59.776393 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:46:59.776636 8208 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 12:46:59.786532 7836 tablet_service.cc:2970] error setting up scanner cc9869bf0a414ed4a1d444d8c7c08019 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.158s (maximum allowed 0.100s): new_scan_request { tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\357\271\324\325\006\"3\n\005slave\022*\n f7080e4ffff44a22a72b8608c75dca3e\020\001\030\001 \001(\001" signature: "<redacted>""\225\227\301p\373\026\334\232\225\005f\212x\357\265\2618\021`0\217\300\347\n\235Q\\t\253cU\353d\3312a\337\313\342\335X\221\266\t\221\034\225jV\301,:\201\203\335\022)\235}P\271\374\270\330" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "dcfcbbc642ea47f3b43d5c78efc7a745"
W20260924 12:46:59.791390 7836 tablet_service.cc:2970] error setting up scanner a5d94a4e61c34ff8b054c60953cefb2c with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.162s (maximum allowed 0.100s): new_scan_request { tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\357\271\324\325\006\"3\n\005slave\022*\n f7080e4ffff44a22a72b8608c75dca3e\020\001\030\001 \001(\001" signature: "<redacted>""\225\227\301p\373\026\334\232\225\005f\212x\357\265\2618\021`0\217\300\347\n\235Q\\t\253cU\353d\3312a\337\313\342\335X\221\266\t\221\034\225jV\301,:\201\203\335\022)\235}P\271\374\270\330" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b1e701df8a3f4c2b8c2f0ef68abb2436"
W20260924 12:46:59.818552 7836 tablet_service.cc:2970] error setting up scanner 1c6b5f1870b340c7816fe8082469cefa with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.190s (maximum allowed 0.100s): new_scan_request { tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\357\271\324\325\006\"3\n\005slave\022*\n f7080e4ffff44a22a72b8608c75dca3e\020\001\030\001 \001(\001" signature: "<redacted>""\225\227\301p\373\026\334\232\225\005f\212x\357\265\2618\021`0\217\300\347\n\235Q\\t\253cU\353d\3312a\337\313\342\335X\221\266\t\221\034\225jV\301,:\201\203\335\022)\235}P\271\374\270\330" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b1e701df8a3f4c2b8c2f0ef68abb2436"
W20260924 12:46:59.819331 7837 tablet_service.cc:2970] error setting up scanner 1927c06cb3384a7fb156928655a93710 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.190s (maximum allowed 0.100s): new_scan_request { tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\357\271\324\325\006\"3\n\005slave\022*\n f7080e4ffff44a22a72b8608c75dca3e\020\001\030\001 \001(\001" signature: "<redacted>""\225\227\301p\373\026\334\232\225\005f\212x\357\265\2618\021`0\217\300\347\n\235Q\\t\253cU\353d\3312a\337\313\342\335X\221\266\t\221\034\225jV\301,:\201\203\335\022)\235}P\271\374\270\330" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "dcfcbbc642ea47f3b43d5c78efc7a745"
W20260924 12:46:59.870028 7836 tablet_service.cc:2970] error setting up scanner 2f6e7707029e47e7a1e9a9cac15b6210 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.241s (maximum allowed 0.100s): new_scan_request { tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\357\271\324\325\006\"3\n\005slave\022*\n f7080e4ffff44a22a72b8608c75dca3e\020\001\030\001 \001(\001" signature: "<redacted>""\225\227\301p\373\026\334\232\225\005f\212x\357\265\2618\021`0\217\300\347\n\235Q\\t\253cU\353d\3312a\337\313\342\335X\221\266\t\221\034\225jV\301,:\201\203\335\022)\235}P\271\374\270\330" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "dcfcbbc642ea47f3b43d5c78efc7a745"
W20260924 12:46:59.870000 7837 tablet_service.cc:2970] error setting up scanner 121ac0810e1e4e8687421ae968239608 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.241s (maximum allowed 0.100s): new_scan_request { tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\357\271\324\325\006\"3\n\005slave\022*\n f7080e4ffff44a22a72b8608c75dca3e\020\001\030\001 \001(\001" signature: "<redacted>""\225\227\301p\373\026\334\232\225\005f\212x\357\265\2618\021`0\217\300\347\n\235Q\\t\253cU\353d\3312a\337\313\342\335X\221\266\t\221\034\225jV\301,:\201\203\335\022)\235}P\271\374\270\330" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b1e701df8a3f4c2b8c2f0ef68abb2436"
W20260924 12:46:59.969024 7837 tablet_service.cc:2970] error setting up scanner 211b2b10d0b54cec99e6a81210496bf7 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.340s (maximum allowed 0.100s): new_scan_request { tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\357\271\324\325\006\"3\n\005slave\022*\n f7080e4ffff44a22a72b8608c75dca3e\020\001\030\001 \001(\001" signature: "<redacted>""\225\227\301p\373\026\334\232\225\005f\212x\357\265\2618\021`0\217\300\347\n\235Q\\t\253cU\353d\3312a\337\313\342\335X\221\266\t\221\034\225jV\301,:\201\203\335\022)\235}P\271\374\270\330" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b1e701df8a3f4c2b8c2f0ef68abb2436"
W20260924 12:46:59.992921 7837 tablet_service.cc:2970] error setting up scanner 74ae6aa3423b438980174ddf3f3fb8d9 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.364s (maximum allowed 0.100s): new_scan_request { tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\357\271\324\325\006\"3\n\005slave\022*\n f7080e4ffff44a22a72b8608c75dca3e\020\001\030\001 \001(\001" signature: "<redacted>""\225\227\301p\373\026\334\232\225\005f\212x\357\265\2618\021`0\217\300\347\n\235Q\\t\253cU\353d\3312a\337\313\342\335X\221\266\t\221\034\225jV\301,:\201\203\335\022)\235}P\271\374\270\330" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "dcfcbbc642ea47f3b43d5c78efc7a745"
W20260924 12:47:00.251814 7837 tablet_service.cc:2970] error setting up scanner fc69729629234455a3f2230381c6fc52 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.623s (maximum allowed 0.100s): new_scan_request { tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\357\271\324\325\006\"3\n\005slave\022*\n f7080e4ffff44a22a72b8608c75dca3e\020\001\030\001 \001(\001" signature: "<redacted>""\225\227\301p\373\026\334\232\225\005f\212x\357\265\2618\021`0\217\300\347\n\235Q\\t\253cU\353d\3312a\337\313\342\335X\221\266\t\221\034\225jV\301,:\201\203\335\022)\235}P\271\374\270\330" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "dcfcbbc642ea47f3b43d5c78efc7a745"
W20260924 12:47:00.275997 7837 tablet_service.cc:2970] error setting up scanner 86177fbad8234560a4911c0ab7056a38 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.647s (maximum allowed 0.100s): new_scan_request { tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\357\271\324\325\006\"3\n\005slave\022*\n f7080e4ffff44a22a72b8608c75dca3e\020\001\030\001 \001(\001" signature: "<redacted>""\225\227\301p\373\026\334\232\225\005f\212x\357\265\2618\021`0\217\300\347\n\235Q\\t\253cU\353d\3312a\337\313\342\335X\221\266\t\221\034\225jV\301,:\201\203\335\022)\235}P\271\374\270\330" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b1e701df8a3f4c2b8c2f0ef68abb2436"
W20260924 12:47:00.463523 8137 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260924 12:47:00.463528 7877 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260924 12:47:00.463629 8137 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20260924 12:47:00.463639 7877 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260924 12:47:00.549523 8237 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:00.549834 8237 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:00.550828 8237 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:00.551425 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:00.551424 8138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:00.552142 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:00.552395 8237 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 12:47:00.727038 7837 tablet_service.cc:2970] error setting up scanner e5ceeec6b2d54f9a84ebb36f18b35355 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.174s (maximum allowed 0.100s): new_scan_request { tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\357\271\324\325\006\"3\n\005slave\022*\n f7080e4ffff44a22a72b8608c75dca3e\020\001\030\001 \001(\001" signature: "<redacted>""\225\227\301p\373\026\334\232\225\005f\212x\357\265\2618\021`0\217\300\347\n\235Q\\t\253cU\353d\3312a\337\313\342\335X\221\266\t\221\034\225jV\301,:\201\203\335\022)\235}P\271\374\270\330" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b1e701df8a3f4c2b8c2f0ef68abb2436"
W20260924 12:47:00.798631 7837 tablet_service.cc:2970] error setting up scanner 659fd93cc578474d8a00330eff6e6845 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.245s (maximum allowed 0.100s): new_scan_request { tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\357\271\324\325\006\"3\n\005slave\022*\n f7080e4ffff44a22a72b8608c75dca3e\020\001\030\001 \001(\001" signature: "<redacted>""\225\227\301p\373\026\334\232\225\005f\212x\357\265\2618\021`0\217\300\347\n\235Q\\t\253cU\353d\3312a\337\313\342\335X\221\266\t\221\034\225jV\301,:\201\203\335\022)\235}P\271\374\270\330" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "dcfcbbc642ea47f3b43d5c78efc7a745"
I20260924 12:47:01.170941 8236 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:01.171212 8236 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:01.172187 8236 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:01.172550 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:01.172919 7876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:01.173561 8073 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:01.174131 8236 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:01.231748 8234 mvcc.cc:205] Tried to move back new op lower bound from 7332880464399323136 to 7332880464189353984. Current Snapshot: MvccSnapshot[applied={T|T < 7332880464399323136 or (T in {7332880464399323136})}]
I20260924 12:47:01.372396 8236 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:01.372627 8236 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:01.373319 8236 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:01.373751 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:01.373805 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:01.374519 8073 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:01.374766 8236 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:01.404223 8247 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=1, attempt_no=0}) took 1782 ms. Trace:
I20260924 12:47:01.404428 8247 rpcz_store.cc:276] 0924 12:46:59.621700 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:46:59.621805 (+ 105us) service_pool.cc:227] Handling call
0924 12:46:59.626060 (+ 4255us) spinlock_profiling.cc:243] Waited 3.49 ms on lock 0x61100006ea58. stack: 00007efde99fc66e 00007efde99fc430 00007efde8a835ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007efdf04cbc73 00007efdf053791c 00007efdf053db25 00007efdf053e54f 00007efdf053e378 00000000009edb7c 00007efde9a20eae 00007efdeccfa6da 00007efde5c9061e
0924 12:47:01.404157 (+1778097us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:00.461793 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:00.461920 (+ 127us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:00.461927 (+ 7us) write_op.cc:435] Acquired schema lock
0924 12:47:00.461929 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:00.462029 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:00.462060 (+ 31us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:00.462065 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:00.462122 (+ 57us) tablet.cc:699] Row locks acquired
0924 12:47:00.462124 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:00.462183 (+ 59us) write_op.cc:270] Start()
0924 12:47:01.231329 (+769146us) write_op.cc:276] Timestamp: P: 1790254020462149 usec, L: 0
0924 12:47:01.231335 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:47:01.231570 (+ 235us) log.cc:842] Serialized 345 byte log entry
0924 12:47:01.402750 (+171180us) op_driver.cc:444] REPLICATION: finished
0924 12:47:01.403276 (+ 526us) write_op.cc:301] APPLY: starting
0924 12:47:01.403296 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:01.403338 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:01.403371 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:01.403495 (+ 124us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:01.403610 (+ 115us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:01.403617 (+ 7us) write_op.cc:312] APPLY: finished
0924 12:47:01.403734 (+ 117us) log.cc:842] Serialized 22 byte log entry
0924 12:47:01.404020 (+ 286us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:01.404044 (+ 24us) write_op.cc:454] Released schema lock
0924 12:47:01.404052 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":8022784,"child_traces":[["op",{"apply.queue_time_us":434,"num_ops":1,"prepare.queue_time_us":835195,"prepare.run_cpu_time_us":805,"prepare.run_wall_time_us":769875,"raft.queue_time_us":305,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":171375,"thread_start_us":321,"threads_started":1}]]}
W20260924 12:47:01.882002 7837 tablet_service.cc:2970] error setting up scanner 0bd717bff5a84b43887a215f96064e0b with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 1.329s (maximum allowed 0.100s): new_scan_request { tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\357\271\324\325\006\"3\n\005slave\022*\n f7080e4ffff44a22a72b8608c75dca3e\020\001\030\001 \001(\001" signature: "<redacted>""\225\227\301p\373\026\334\232\225\005f\212x\357\265\2618\021`0\217\300\347\n\235Q\\t\253cU\353d\3312a\337\313\342\335X\221\266\t\221\034\225jV\301,:\201\203\335\022)\235}P\271\374\270\330" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b1e701df8a3f4c2b8c2f0ef68abb2436"
I20260924 12:47:01.949914 8246 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:01.950142 8246 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:01.951022 8246 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:01.951520 8138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:01.951511 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:01.952304 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:02.093022 8246 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:02.123680 8253 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=2, attempt_no=0}) took 1550 ms. Trace:
I20260924 12:47:02.123834 8253 rpcz_store.cc:276] 0924 12:47:00.573276 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:00.573341 (+ 65us) service_pool.cc:227] Handling call
0924 12:47:02.123621 (+1550280us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:01.231991 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:01.232111 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:01.232118 (+ 7us) write_op.cc:435] Acquired schema lock
0924 12:47:01.232120 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:01.232238 (+ 118us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:01.232250 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:01.232255 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:01.232313 (+ 58us) tablet.cc:699] Row locks acquired
0924 12:47:01.232315 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:01.232366 (+ 51us) write_op.cc:270] Start()
0924 12:47:01.906520 (+674154us) write_op.cc:276] Timestamp: P: 1790254021232314 usec, L: 0
0924 12:47:01.906526 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:47:01.906745 (+ 219us) log.cc:842] Serialized 354 byte log entry
0924 12:47:02.122505 (+215760us) op_driver.cc:444] REPLICATION: finished
0924 12:47:02.123025 (+ 520us) write_op.cc:301] APPLY: starting
0924 12:47:02.123041 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:02.123076 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:02.123101 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:02.123172 (+ 71us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:02.123200 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:02.123204 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:47:02.123289 (+ 85us) log.cc:842] Serialized 22 byte log entry
0924 12:47:02.123490 (+ 201us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:02.123510 (+ 20us) write_op.cc:454] Released schema lock
0924 12:47:02.123518 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":458,"num_ops":1,"prepare.queue_time_us":658020,"prepare.run_cpu_time_us":757,"prepare.run_wall_time_us":674831,"replication_time_us":215919,"thread_start_us":312,"threads_started":1}]]}
I20260924 12:47:02.589970 8248 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:02.590186 8248 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:02.591205 8248 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:02.591615 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:02.591869 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:02.592458 8073 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
W20260924 12:47:03.017668 7837 tablet_service.cc:2970] error setting up scanner 86f0393e182c44f7a8f07d89a70703df with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1790254022017284 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1790254021232314 usec, L: 0 Physical time difference: 0.785s: new_scan_request { tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\357\271\324\325\006\"3\n\005slave\022*\n f7080e4ffff44a22a72b8608c75dca3e\020\001\030\001 \001(\001" signature: "<redacted>""\225\227\301p\373\026\334\232\225\005f\212x\357\265\2618\021`0\217\300\347\n\235Q\\t\253cU\353d\3312a\337\313\342\335X\221\266\t\221\034\225jV\301,:\201\203\335\022)\235}P\271\374\270\330" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "dcfcbbc642ea47f3b43d5c78efc7a745"
I20260924 12:47:03.018141 7837 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:49042 (request call id 15) took 1001 ms. Trace:
I20260924 12:47:03.018271 7837 rpcz_store.cc:276] 0924 12:47:02.016324 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:02.016411 (+ 87us) service_pool.cc:227] Handling call
0924 12:47:02.016629 (+ 218us) tablet_service.cc:2780] Created scanner 86f0393e182c44f7a8f07d89a70703df for tablet 0121d1741e224a7ba39ced9c1fdb2b4b, query id is dcfcbbc642ea47f3b43d5c78efc7a745
0924 12:47:02.017310 (+ 681us) tablet_service.cc:2920] Creating iterator
0924 12:47:02.017339 (+ 29us) tablet_service.cc:3297] Waiting safe time to advance
0924 12:47:03.017593 (+1000254us) tablet_service.cc:2945] Iterator created
0924 12:47:03.017608 (+ 15us) tablet_service.cc:2967] Iterator init: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1790254022017284 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1790254021232314 usec, L: 0 Physical time difference: 0.785s
0924 12:47:03.018112 (+ 504us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260924 12:47:03.083518 8248 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:03.086916 8258 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=3, attempt_no=0}) took 1679 ms. Trace:
I20260924 12:47:03.087033 8258 rpcz_store.cc:276] 0924 12:47:01.407838 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:01.407889 (+ 51us) service_pool.cc:227] Handling call
0924 12:47:03.086862 (+1678973us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:01.906886 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:01.907009 (+ 123us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:01.907015 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:47:01.907016 (+ 1us) tablet.cc:660] Decoding operations
0924 12:47:01.907092 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:01.907101 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:01.907104 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:01.907149 (+ 45us) tablet.cc:699] Row locks acquired
0924 12:47:01.907151 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:01.907199 (+ 48us) write_op.cc:270] Start()
0924 12:47:02.447349 (+540150us) write_op.cc:276] Timestamp: P: 1790254021907137 usec, L: 0
0924 12:47:02.447354 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:47:02.447598 (+ 244us) log.cc:842] Serialized 354 byte log entry
0924 12:47:03.085844 (+638246us) op_driver.cc:444] REPLICATION: finished
0924 12:47:03.086279 (+ 435us) write_op.cc:301] APPLY: starting
0924 12:47:03.086293 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:03.086321 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:03.086341 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:03.086400 (+ 59us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:03.086425 (+ 25us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:03.086429 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:47:03.086533 (+ 104us) log.cc:842] Serialized 22 byte log entry
0924 12:47:03.086641 (+ 108us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:03.086654 (+ 13us) write_op.cc:454] Released schema lock
0924 12:47:03.086661 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":377,"num_ops":1,"prepare.queue_time_us":498582,"prepare.run_cpu_time_us":776,"prepare.run_wall_time_us":540844,"raft.queue_time_us":78,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":638465,"spinlock_wait_cycles":9984,"thread_start_us":286,"threads_started":1}]]}
I20260924 12:47:03.139746 8246 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:03.139914 8246 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:03.140628 8246 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:03.141048 8138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:03.141539 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:03.142151 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:03.167127 8246 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:03.207948 8258 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=4, attempt_no=0}) took 1082 ms. Trace:
I20260924 12:47:03.208107 8258 rpcz_store.cc:276] 0924 12:47:02.125498 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:02.125543 (+ 45us) service_pool.cc:227] Handling call
0924 12:47:03.207869 (+1082326us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:02.447743 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:02.447875 (+ 132us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:02.447883 (+ 8us) write_op.cc:435] Acquired schema lock
0924 12:47:02.447885 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:02.447973 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:02.447985 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:02.447989 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:02.448044 (+ 55us) tablet.cc:699] Row locks acquired
0924 12:47:02.448046 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:02.448098 (+ 52us) write_op.cc:270] Start()
0924 12:47:02.972268 (+524170us) write_op.cc:276] Timestamp: P: 1790254022448030 usec, L: 0
0924 12:47:02.972273 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:47:02.972519 (+ 246us) log.cc:842] Serialized 354 byte log entry
0924 12:47:03.207254 (+234735us) op_driver.cc:444] REPLICATION: finished
0924 12:47:03.207417 (+ 163us) write_op.cc:301] APPLY: starting
0924 12:47:03.207436 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:03.207471 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:03.207491 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:03.207553 (+ 62us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:03.207581 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:03.207585 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:47:03.207656 (+ 71us) log.cc:842] Serialized 22 byte log entry
0924 12:47:03.207744 (+ 88us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:03.207764 (+ 20us) write_op.cc:454] Released schema lock
0924 12:47:03.207772 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":82,"num_ops":1,"prepare.queue_time_us":321719,"prepare.run_cpu_time_us":782,"prepare.run_wall_time_us":524851,"replication_time_us":234945}]]}
I20260924 12:47:03.866650 8248 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:03.866853 8248 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:03.867601 8248 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:03.868069 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:03.868139 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:03.868708 8073 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:03.900101 8246 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:03.900317 8246 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:03.901029 8246 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:03.901515 8138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:03.901793 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:03.902273 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:03.902544 8246 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 12:47:04.316824 7837 tablet_service.cc:2970] error setting up scanner 13ca0e3e3e87427c98eb3a7e6f4cbbc3 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.414s (maximum allowed 0.100s): new_scan_request { tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\357\271\324\325\006\"3\n\005slave\022*\n f7080e4ffff44a22a72b8608c75dca3e\020\001\030\001 \001(\001" signature: "<redacted>""\225\227\301p\373\026\334\232\225\005f\212x\357\265\2618\021`0\217\300\347\n\235Q\\t\253cU\353d\3312a\337\313\342\335X\221\266\t\221\034\225jV\301,:\201\203\335\022)\235}P\271\374\270\330" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b1e701df8a3f4c2b8c2f0ef68abb2436"
I20260924 12:47:04.507718 8248 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:04.524366 8272 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=6, attempt_no=0}) took 1314 ms. Trace:
I20260924 12:47:04.524513 8272 rpcz_store.cc:276] 0924 12:47:03.209817 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:03.209875 (+ 58us) service_pool.cc:227] Handling call
0924 12:47:04.524307 (+1314432us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:03.441239 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:03.441365 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:03.441371 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:47:03.441374 (+ 3us) tablet.cc:660] Decoding operations
0924 12:47:03.441467 (+ 93us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:03.441479 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:03.441483 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:03.441543 (+ 60us) tablet.cc:699] Row locks acquired
0924 12:47:03.441545 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:03.441590 (+ 45us) write_op.cc:270] Start()
0924 12:47:03.758759 (+317169us) write_op.cc:276] Timestamp: P: 1790254023441523 usec, L: 0
0924 12:47:03.758765 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:47:03.759038 (+ 273us) log.cc:842] Serialized 354 byte log entry
0924 12:47:04.523207 (+764169us) op_driver.cc:444] REPLICATION: finished
0924 12:47:04.523696 (+ 489us) write_op.cc:301] APPLY: starting
0924 12:47:04.523714 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:04.523755 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:04.523785 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:04.523920 (+ 135us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:04.523959 (+ 39us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:04.523964 (+ 5us) write_op.cc:312] APPLY: finished
0924 12:47:04.524086 (+ 122us) log.cc:842] Serialized 22 byte log entry
0924 12:47:04.524183 (+ 97us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:04.524202 (+ 19us) write_op.cc:454] Released schema lock
0924 12:47:04.524209 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":417,"num_ops":1,"prepare.queue_time_us":230956,"prepare.run_cpu_time_us":810,"prepare.run_wall_time_us":317874,"replication_time_us":764405,"thread_start_us":275,"threads_started":1}]]}
W20260924 12:47:04.557404 7837 tablet_service.cc:2970] error setting up scanner 89dbf3c3c0a94ccba1db1ca6e0d775a0 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.654s (maximum allowed 0.100s): new_scan_request { tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\357\271\324\325\006\"3\n\005slave\022*\n f7080e4ffff44a22a72b8608c75dca3e\020\001\030\001 \001(\001" signature: "<redacted>""\225\227\301p\373\026\334\232\225\005f\212x\357\265\2618\021`0\217\300\347\n\235Q\\t\253cU\353d\3312a\337\313\342\335X\221\266\t\221\034\225jV\301,:\201\203\335\022)\235}P\271\374\270\330" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "dcfcbbc642ea47f3b43d5c78efc7a745"
I20260924 12:47:04.846230 8246 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:04.846434 8246 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:04.847236 8246 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:04.847661 8138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:04.847882 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:04.848414 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:05.241766 8248 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:05.241988 8248 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:05.242867 8248 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:05.243937 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:05.243952 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:05.245579 8074 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:05.246073 8248 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:05.278566 8279 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=7, attempt_no=0}) took 1327 ms. Trace:
I20260924 12:47:05.278712 8279 rpcz_store.cc:276] 0924 12:47:03.950674 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:03.950737 (+ 63us) service_pool.cc:227] Handling call
0924 12:47:05.278506 (+1327769us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:03.951243 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:03.951339 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:03.951343 (+ 4us) write_op.cc:435] Acquired schema lock
0924 12:47:03.951344 (+ 1us) tablet.cc:660] Decoding operations
0924 12:47:03.951462 (+ 118us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:03.951473 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:03.951478 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:03.951525 (+ 47us) tablet.cc:699] Row locks acquired
0924 12:47:03.951527 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:03.951561 (+ 34us) write_op.cc:270] Start()
0924 12:47:04.452707 (+501146us) write_op.cc:276] Timestamp: P: 1790254023951511 usec, L: 0
0924 12:47:04.452712 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:47:04.452939 (+ 227us) log.cc:842] Serialized 354 byte log entry
0924 12:47:05.277431 (+824492us) op_driver.cc:444] REPLICATION: finished
0924 12:47:05.277926 (+ 495us) write_op.cc:301] APPLY: starting
0924 12:47:05.277945 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:05.277981 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:05.278012 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:05.278092 (+ 80us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:05.278120 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:05.278124 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:47:05.278224 (+ 100us) log.cc:842] Serialized 22 byte log entry
0924 12:47:05.278321 (+ 97us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:05.278342 (+ 21us) write_op.cc:454] Released schema lock
0924 12:47:05.278350 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":427,"num_ops":1,"prepare.queue_time_us":53,"prepare.run_cpu_time_us":744,"prepare.run_wall_time_us":501774,"replication_time_us":824672,"thread_start_us":244,"threads_started":1}]]}
I20260924 12:47:05.528597 8248 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:05.528869 8248 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:05.529666 8248 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:05.530072 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:05.530179 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:05.530869 8073 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:05.531134 8248 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:05.538316 8279 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=8, attempt_no=0}) took 1011 ms. Trace:
I20260924 12:47:05.538432 8279 rpcz_store.cc:276] 0924 12:47:04.526374 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:04.526437 (+ 63us) service_pool.cc:227] Handling call
0924 12:47:05.538257 (+1011820us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:04.526979 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:04.527084 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:04.527089 (+ 5us) write_op.cc:435] Acquired schema lock
0924 12:47:04.527090 (+ 1us) tablet.cc:660] Decoding operations
0924 12:47:04.527165 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:04.527174 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:04.527177 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:04.527220 (+ 43us) tablet.cc:699] Row locks acquired
0924 12:47:04.527221 (+ 1us) write_op.cc:260] PREPARE: finished
0924 12:47:04.527256 (+ 35us) write_op.cc:270] Start()
0924 12:47:04.588428 (+ 61172us) write_op.cc:276] Timestamp: P: 1790254024527190 usec, L: 0
0924 12:47:04.588433 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:47:04.588671 (+ 238us) log.cc:842] Serialized 354 byte log entry
0924 12:47:05.537637 (+948966us) op_driver.cc:444] REPLICATION: finished
0924 12:47:05.537757 (+ 120us) write_op.cc:301] APPLY: starting
0924 12:47:05.537774 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:05.537808 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:05.537851 (+ 43us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:05.537926 (+ 75us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:05.537958 (+ 32us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:05.537962 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:47:05.538064 (+ 102us) log.cc:842] Serialized 22 byte log entry
0924 12:47:05.538147 (+ 83us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:05.538166 (+ 19us) write_op.cc:454] Released schema lock
0924 12:47:05.538174 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":43,"num_ops":1,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":700,"prepare.run_wall_time_us":61777,"replication_time_us":949172}]]}
I20260924 12:47:06.479769 8246 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:06.480239 7876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.5.77.194:39441 (request call id 1741) took 1632 ms. Trace:
I20260924 12:47:06.480386 7876 rpcz_store.cc:276] 0924 12:47:04.847671 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:04.847741 (+ 70us) service_pool.cc:227] Handling call
0924 12:47:04.847942 (+ 201us) raft_consensus.cc:1457] Updating replica for 2 ops
0924 12:47:04.848068 (+ 126us) raft_consensus.cc:1520] Early marking committed up to index 8
0924 12:47:04.848484 (+ 416us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0924 12:47:04.848629 (+ 145us) write_op.cc:270] Start()
0924 12:47:05.488801 (+640172us) write_op.cc:276] Timestamp: P: 1790254023951511 usec, L: 0
0924 12:47:05.489486 (+ 685us) write_op.cc:270] Start()
0924 12:47:06.478653 (+989167us) write_op.cc:276] Timestamp: P: 1790254024527190 usec, L: 0
0924 12:47:06.479314 (+ 661us) log.cc:842] Serialized 708 byte log entry
0924 12:47:06.479648 (+ 334us) raft_consensus.cc:1631] Marking committed up to 8
0924 12:47:06.479657 (+ 9us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 12:47:06.479694 (+ 37us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 12:47:06.480142 (+ 448us) raft_consensus.cc:1671] finished
0924 12:47:06.480145 (+ 3us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 12:47:06.480198 (+ 53us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:05.489316 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:05.489452 (+ 136us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:05.489460 (+ 8us) write_op.cc:435] Acquired schema lock
0924 12:47:05.489463 (+ 3us) tablet.cc:660] Decoding operations
0924 12:47:05.489575 (+ 112us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:05.489589 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:05.489592 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:05.489653 (+ 61us) tablet.cc:699] Row locks acquired
0924 12:47:05.489655 (+ 2us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0924 12:47:06.479162 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:06.479282 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:06.479288 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:47:06.479289 (+ 1us) tablet.cc:660] Decoding operations
0924 12:47:06.479368 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:06.479400 (+ 32us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:06.479403 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:06.479454 (+ 51us) tablet.cc:699] Row locks acquired
0924 12:47:06.479457 (+ 3us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":241,"threads_started":1,"wal-append.queue_time_us":327,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":357,"prepare.run_cpu_time_us":446,"prepare.run_wall_time_us":446,"thread_start_us":233,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":25,"prepare.queue_time_us":363,"prepare.run_cpu_time_us":403,"prepare.run_wall_time_us":403,"thread_start_us":227,"threads_started":1}]]}
I20260924 12:47:06.945323 8248 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:06.945564 8248 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:06.946372 8248 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:06.947019 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:06.947048 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:06.947623 8074 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:06.984297 8291 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=9, attempt_no=0}) took 1703 ms. Trace:
I20260924 12:47:06.984459 8291 rpcz_store.cc:276] 0924 12:47:05.280376 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:05.280450 (+ 74us) service_pool.cc:227] Handling call
0924 12:47:06.984232 (+1703782us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:05.281168 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:05.281317 (+ 149us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:05.281321 (+ 4us) write_op.cc:435] Acquired schema lock
0924 12:47:05.281323 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:05.281431 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:05.281441 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:05.281444 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:05.281493 (+ 49us) tablet.cc:699] Row locks acquired
0924 12:47:05.281494 (+ 1us) write_op.cc:260] PREPARE: finished
0924 12:47:05.281551 (+ 57us) write_op.cc:270] Start()
0924 12:47:05.964716 (+683165us) write_op.cc:276] Timestamp: P: 1790254025281510 usec, L: 0
0924 12:47:05.964721 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:47:05.964984 (+ 263us) log.cc:842] Serialized 354 byte log entry
0924 12:47:06.982942 (+1017958us) op_driver.cc:444] REPLICATION: finished
0924 12:47:06.983524 (+ 582us) write_op.cc:301] APPLY: starting
0924 12:47:06.983543 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:06.983585 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:06.983620 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:06.983732 (+ 112us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:06.983766 (+ 34us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:06.983772 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:47:06.983969 (+ 197us) log.cc:842] Serialized 22 byte log entry
0924 12:47:06.984072 (+ 103us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:06.984094 (+ 22us) write_op.cc:454] Released schema lock
0924 12:47:06.984102 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":516,"mutex_wait_us":59,"num_ops":1,"prepare.queue_time_us":318,"prepare.run_cpu_time_us":896,"prepare.run_wall_time_us":683991,"raft.queue_time_us":159,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":1018181,"spinlock_wait_cycles":98688,"thread_start_us":589,"threads_started":2}]]}
I20260924 12:47:07.041364 8246 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:07.041568 8246 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:07.042275 8246 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:07.042876 8138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:07.042891 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:07.043545 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:07.379699 8246 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:07.423559 8291 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=10, attempt_no=0}) took 1883 ms. Trace:
I20260924 12:47:07.423691 8291 rpcz_store.cc:276] 0924 12:47:05.540096 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:05.540161 (+ 65us) service_pool.cc:227] Handling call
0924 12:47:07.423476 (+1883315us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:05.965193 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:05.965325 (+ 132us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:05.965331 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:47:05.965332 (+ 1us) tablet.cc:660] Decoding operations
0924 12:47:05.965408 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:05.965418 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:05.965421 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:05.965466 (+ 45us) tablet.cc:699] Row locks acquired
0924 12:47:05.965468 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:05.965521 (+ 53us) write_op.cc:270] Start()
0924 12:47:06.539669 (+574148us) write_op.cc:276] Timestamp: P: 1790254025965451 usec, L: 0
0924 12:47:06.539674 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:47:06.539915 (+ 241us) log.cc:842] Serialized 354 byte log entry
0924 12:47:07.422764 (+882849us) op_driver.cc:444] REPLICATION: finished
0924 12:47:07.422933 (+ 169us) write_op.cc:301] APPLY: starting
0924 12:47:07.422954 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:07.422992 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:07.423029 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:07.423104 (+ 75us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:07.423134 (+ 30us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:07.423138 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:47:07.423225 (+ 87us) log.cc:842] Serialized 22 byte log entry
0924 12:47:07.423326 (+ 101us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:07.423348 (+ 22us) write_op.cc:454] Released schema lock
0924 12:47:07.423356 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":94,"num_ops":1,"prepare.queue_time_us":424558,"prepare.run_cpu_time_us":754,"prepare.run_wall_time_us":574806,"replication_time_us":883052}]]}
I20260924 12:47:07.432453 8246 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:07.432652 8246 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:07.433348 8246 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:07.433846 8138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:07.433974 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:07.434724 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:07.434984 8246 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:07.508245 8248 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:08.155094 8248 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:08.155292 8248 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:08.156067 8248 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:08.156741 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:08.156735 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:08.157312 8074 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:08.354167 8246 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:08.354413 8246 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:08.355139 8246 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:08.355623 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:08.355600 8138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:08.356225 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:08.356494 8246 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:08.649773 8248 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 12:47:09.400350 7837 tablet_service.cc:2970] error setting up scanner 82aa2946c96a4355a562e9022893b3e3 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1790254028400036 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1790254028381644 usec, L: 0 Physical time difference: 0.018s: new_scan_request { tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\357\271\324\325\006\"3\n\005slave\022*\n f7080e4ffff44a22a72b8608c75dca3e\020\001\030\001 \001(\001" signature: "<redacted>""\225\227\301p\373\026\334\232\225\005f\212x\357\265\2618\021`0\217\300\347\n\235Q\\t\253cU\353d\3312a\337\313\342\335X\221\266\t\221\034\225jV\301,:\201\203\335\022)\235}P\271\374\270\330" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "dcfcbbc642ea47f3b43d5c78efc7a745"
I20260924 12:47:09.400866 7837 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:49042 (request call id 18) took 1001 ms. Trace:
I20260924 12:47:09.400983 7837 rpcz_store.cc:276] 0924 12:47:08.398943 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:08.399008 (+ 65us) service_pool.cc:227] Handling call
0924 12:47:08.399222 (+ 214us) tablet_service.cc:2780] Created scanner 82aa2946c96a4355a562e9022893b3e3 for tablet 0121d1741e224a7ba39ced9c1fdb2b4b, query id is dcfcbbc642ea47f3b43d5c78efc7a745
0924 12:47:08.400035 (+ 813us) tablet_service.cc:2920] Creating iterator
0924 12:47:08.400071 (+ 36us) tablet_service.cc:3297] Waiting safe time to advance
0924 12:47:09.400289 (+1000218us) tablet_service.cc:2945] Iterator created
0924 12:47:09.400304 (+ 15us) tablet_service.cc:2967] Iterator init: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1790254028400036 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1790254028381644 usec, L: 0 Physical time difference: 0.018s
0924 12:47:09.400798 (+ 494us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260924 12:47:09.415776 7836 tablet_service.cc:2970] error setting up scanner d73ba2dd2309467ca07096ba54b78768 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1790254028415494 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1790254028381644 usec, L: 0 Physical time difference: 0.034s: new_scan_request { tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\357\271\324\325\006\"3\n\005slave\022*\n f7080e4ffff44a22a72b8608c75dca3e\020\001\030\001 \001(\001" signature: "<redacted>""\225\227\301p\373\026\334\232\225\005f\212x\357\265\2618\021`0\217\300\347\n\235Q\\t\253cU\353d\3312a\337\313\342\335X\221\266\t\221\034\225jV\301,:\201\203\335\022)\235}P\271\374\270\330" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b1e701df8a3f4c2b8c2f0ef68abb2436"
I20260924 12:47:09.416253 7836 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:49042 (request call id 19) took 1001 ms. Trace:
I20260924 12:47:09.416368 7836 rpcz_store.cc:276] 0924 12:47:08.414354 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:08.414404 (+ 50us) service_pool.cc:227] Handling call
0924 12:47:08.414642 (+ 238us) tablet_service.cc:2780] Created scanner d73ba2dd2309467ca07096ba54b78768 for tablet 0121d1741e224a7ba39ced9c1fdb2b4b, query id is b1e701df8a3f4c2b8c2f0ef68abb2436
0924 12:47:08.415495 (+ 853us) tablet_service.cc:2920] Creating iterator
0924 12:47:08.415520 (+ 25us) tablet_service.cc:3297] Waiting safe time to advance
0924 12:47:09.415733 (+1000213us) tablet_service.cc:2945] Iterator created
0924 12:47:09.415748 (+ 15us) tablet_service.cc:2967] Iterator init: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1790254028415494 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1790254028381644 usec, L: 0 Physical time difference: 0.034s
0924 12:47:09.416195 (+ 447us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260924 12:47:09.440764 8246 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:09.440979 8246 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:09.441735 8246 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:09.442312 8138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:09.442503 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:09.443146 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:09.443434 8246 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:09.481583 8248 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:09.481807 8248 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:09.482625 8248 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:09.483249 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:09.483289 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:09.483872 8309 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=14, attempt_no=0}) took 1102 ms. Trace:
I20260924 12:47:09.483875 8074 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:09.484005 8309 rpcz_store.cc:276] 0924 12:47:08.381258 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:08.381323 (+ 65us) service_pool.cc:227] Handling call
0924 12:47:09.483822 (+1102499us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:08.381818 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:08.381931 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:08.381936 (+ 5us) write_op.cc:435] Acquired schema lock
0924 12:47:08.381939 (+ 3us) tablet.cc:660] Decoding operations
0924 12:47:08.382019 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:08.382029 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:08.382032 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:08.382074 (+ 42us) tablet.cc:699] Row locks acquired
0924 12:47:08.382076 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:08.382106 (+ 30us) write_op.cc:270] Start()
0924 12:47:08.564250 (+182144us) write_op.cc:276] Timestamp: P: 1790254028382053 usec, L: 0
0924 12:47:08.564256 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:47:08.564537 (+ 281us) log.cc:842] Serialized 354 byte log entry
0924 12:47:09.481923 (+917386us) op_driver.cc:444] REPLICATION: finished
0924 12:47:09.482713 (+ 790us) write_op.cc:301] APPLY: starting
0924 12:47:09.482731 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:09.482765 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:09.482800 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:09.483012 (+ 212us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:09.483049 (+ 37us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:09.483055 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:47:09.483641 (+ 586us) log.cc:842] Serialized 22 byte log entry
0924 12:47:09.483711 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:09.483726 (+ 15us) write_op.cc:454] Released schema lock
0924 12:47:09.483733 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":724,"num_ops":1,"prepare.queue_time_us":42,"prepare.run_cpu_time_us":746,"prepare.run_wall_time_us":182801,"replication_time_us":917608,"thread_start_us":616,"threads_started":1}]]}
I20260924 12:47:09.637013 8248 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:09.687834 8309 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=15, attempt_no=0}) took 1280 ms. Trace:
I20260924 12:47:09.687988 8309 rpcz_store.cc:276] 0924 12:47:08.407293 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:08.407365 (+ 72us) service_pool.cc:227] Handling call
0924 12:47:09.687759 (+1280394us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:08.564654 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:08.564782 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:08.564789 (+ 7us) write_op.cc:435] Acquired schema lock
0924 12:47:08.564791 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:08.564872 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:08.564883 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:08.564887 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:08.564931 (+ 44us) tablet.cc:699] Row locks acquired
0924 12:47:08.564933 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:08.564971 (+ 38us) write_op.cc:270] Start()
0924 12:47:09.183130 (+618159us) write_op.cc:276] Timestamp: P: 1790254028564901 usec, L: 0
0924 12:47:09.183136 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:47:09.183380 (+ 244us) log.cc:842] Serialized 354 byte log entry
0924 12:47:09.687121 (+503741us) op_driver.cc:444] REPLICATION: finished
0924 12:47:09.687230 (+ 109us) write_op.cc:301] APPLY: starting
0924 12:47:09.687252 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:09.687294 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:09.687319 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:09.687419 (+ 100us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:09.687453 (+ 34us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:09.687457 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:47:09.687536 (+ 79us) log.cc:842] Serialized 22 byte log entry
0924 12:47:09.687631 (+ 95us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:09.687652 (+ 21us) write_op.cc:454] Released schema lock
0924 12:47:09.687659 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":42,"num_ops":1,"prepare.queue_time_us":156797,"prepare.run_cpu_time_us":731,"prepare.run_wall_time_us":618798,"replication_time_us":503955}]]}
I20260924 12:47:10.405830 8246 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:10.406028 8246 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:10.406780 8246 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:10.407323 8138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:10.407686 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:10.408257 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:10.582953 8311 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:10.583279 8311 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:10.584121 8311 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:10.584587 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:10.584596 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:10.585198 8073 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:10.585570 8311 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:10.615365 8316 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=16, attempt_no=0}) took 1129 ms. Trace:
I20260924 12:47:10.615530 8316 rpcz_store.cc:276] 0924 12:47:09.485656 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:09.485719 (+ 63us) service_pool.cc:227] Handling call
0924 12:47:10.615309 (+1129590us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:09.486159 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:09.486247 (+ 88us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:09.486251 (+ 4us) write_op.cc:435] Acquired schema lock
0924 12:47:09.486253 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:09.486320 (+ 67us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:09.486329 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:09.486332 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:09.486369 (+ 37us) tablet.cc:699] Row locks acquired
0924 12:47:09.486371 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:09.486407 (+ 36us) write_op.cc:270] Start()
0924 12:47:10.387579 (+901172us) write_op.cc:276] Timestamp: P: 1790254029486354 usec, L: 0
0924 12:47:10.387585 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:47:10.387826 (+ 241us) log.cc:842] Serialized 354 byte log entry
0924 12:47:10.614195 (+226369us) op_driver.cc:444] REPLICATION: finished
0924 12:47:10.614703 (+ 508us) write_op.cc:301] APPLY: starting
0924 12:47:10.614723 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:10.614772 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:10.614819 (+ 47us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:10.614909 (+ 90us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:10.614952 (+ 43us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:10.614959 (+ 7us) write_op.cc:312] APPLY: finished
0924 12:47:10.615078 (+ 119us) log.cc:842] Serialized 22 byte log entry
0924 12:47:10.615187 (+ 109us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:10.615207 (+ 20us) write_op.cc:454] Released schema lock
0924 12:47:10.615214 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":445,"num_ops":1,"prepare.queue_time_us":56,"prepare.run_cpu_time_us":689,"prepare.run_wall_time_us":901771,"raft.queue_time_us":42,"raft.run_cpu_time_us":269,"raft.run_wall_time_us":268,"replication_time_us":226579,"spinlock_wait_cycles":91392,"thread_start_us":314,"threads_started":1}]]}
I20260924 12:47:11.011143 8246 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:11.556751 8246 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:11.556982 8246 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:11.557734 8246 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:11.558362 8138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:11.558574 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:11.559196 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:11.560074 8246 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:11.561743 8321 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=17, attempt_no=0}) took 1872 ms. Trace:
I20260924 12:47:11.561900 8321 rpcz_store.cc:276] 0924 12:47:09.689699 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:09.689758 (+ 59us) service_pool.cc:227] Handling call
0924 12:47:11.561666 (+1871908us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:10.387953 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:10.388087 (+ 134us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:10.388094 (+ 7us) write_op.cc:435] Acquired schema lock
0924 12:47:10.388096 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:10.388196 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:10.388208 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:10.388212 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:10.388269 (+ 57us) tablet.cc:699] Row locks acquired
0924 12:47:10.388272 (+ 3us) write_op.cc:260] PREPARE: finished
0924 12:47:10.388323 (+ 51us) write_op.cc:270] Start()
0924 12:47:11.366505 (+978182us) write_op.cc:276] Timestamp: P: 1790254030388273 usec, L: 0
0924 12:47:11.366511 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:47:11.366793 (+ 282us) log.cc:842] Serialized 354 byte log entry
0924 12:47:11.558246 (+191453us) op_driver.cc:444] REPLICATION: finished
0924 12:47:11.561103 (+ 2857us) write_op.cc:301] APPLY: starting
0924 12:47:11.561118 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:11.561151 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:11.561183 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:11.561268 (+ 85us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:11.561298 (+ 30us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:11.561303 (+ 5us) write_op.cc:312] APPLY: finished
0924 12:47:11.561442 (+ 139us) log.cc:842] Serialized 22 byte log entry
0924 12:47:11.561546 (+ 104us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:11.561563 (+ 17us) write_op.cc:454] Released schema lock
0924 12:47:11.561569 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2796,"num_ops":1,"prepare.queue_time_us":697751,"prepare.run_cpu_time_us":867,"prepare.run_wall_time_us":978937,"raft.queue_time_us":28,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":182,"replication_time_us":191705,"spinlock_wait_cycles":147456,"thread_start_us":2695,"threads_started":1}]]}
I20260924 12:47:11.679984 8319 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:11.680282 8319 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:11.681192 8319 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:11.681452 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:11.682169 7876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:11.682925 8073 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:11.683290 8319 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 12:47:11.963757 7836 tablet_service.cc:2970] error setting up scanner 5ef34623303b47aabbe5b34983310b99 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.405s (maximum allowed 0.100s): new_scan_request { tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\357\271\324\325\006\"3\n\005slave\022*\n f7080e4ffff44a22a72b8608c75dca3e\020\001\030\001 \001(\001" signature: "<redacted>""\225\227\301p\373\026\334\232\225\005f\212x\357\265\2618\021`0\217\300\347\n\235Q\\t\253cU\353d\3312a\337\313\342\335X\221\266\t\221\034\225jV\301,:\201\203\335\022)\235}P\271\374\270\330" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "dcfcbbc642ea47f3b43d5c78efc7a745"
I20260924 12:47:12.014554 8246 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:12.014740 8246 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:12.015383 8246 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:12.015776 8137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:12.015820 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:12.016332 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:12.016624 8246 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:12.672879 8319 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:12.673077 8319 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:12.673857 8319 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:12.674317 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:12.674396 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:12.674940 8073 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:12.675336 8319 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:12.703190 8331 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=18, attempt_no=0}) took 2086 ms. Trace:
I20260924 12:47:12.703347 8331 rpcz_store.cc:276] 0924 12:47:10.617175 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:10.617249 (+ 74us) service_pool.cc:227] Handling call
0924 12:47:12.703143 (+2085894us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:11.366932 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:11.367051 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:11.367058 (+ 7us) write_op.cc:435] Acquired schema lock
0924 12:47:11.367060 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:11.367156 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:11.367168 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:11.367172 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:11.367227 (+ 55us) tablet.cc:699] Row locks acquired
0924 12:47:11.367230 (+ 3us) write_op.cc:260] PREPARE: finished
0924 12:47:11.367280 (+ 50us) write_op.cc:270] Start()
0924 12:47:12.025433 (+658153us) write_op.cc:276] Timestamp: P: 1790254031367234 usec, L: 0
0924 12:47:12.025439 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:47:12.025670 (+ 231us) log.cc:842] Serialized 354 byte log entry
0924 12:47:12.702058 (+676388us) op_driver.cc:444] REPLICATION: finished
0924 12:47:12.702576 (+ 518us) write_op.cc:301] APPLY: starting
0924 12:47:12.702593 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:12.702630 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:12.702656 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:12.702726 (+ 70us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:12.702754 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:12.702758 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:47:12.702829 (+ 71us) log.cc:842] Serialized 22 byte log entry
0924 12:47:12.702928 (+ 99us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:12.702948 (+ 20us) write_op.cc:454] Released schema lock
0924 12:47:12.702956 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":459,"num_ops":1,"prepare.queue_time_us":749251,"prepare.run_cpu_time_us":760,"prepare.run_wall_time_us":658828,"raft.queue_time_us":36,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":198,"replication_time_us":676589,"thread_start_us":359,"threads_started":1}]]}
I20260924 12:47:12.922039 8319 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:12.922238 8319 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:12.923012 8319 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:12.923525 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:12.923508 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:12.924153 8073 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:12.924530 8319 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:12.970892 8331 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=19, attempt_no=0}) took 1407 ms. Trace:
I20260924 12:47:12.971021 8331 rpcz_store.cc:276] 0924 12:47:11.563745 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:11.563824 (+ 79us) service_pool.cc:227] Handling call
0924 12:47:12.970832 (+1407008us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:12.025814 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:12.025934 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:12.025941 (+ 7us) write_op.cc:435] Acquired schema lock
0924 12:47:12.025943 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:12.026047 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:12.026059 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:12.026064 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:12.026132 (+ 68us) tablet.cc:699] Row locks acquired
0924 12:47:12.026135 (+ 3us) write_op.cc:260] PREPARE: finished
0924 12:47:12.026185 (+ 50us) write_op.cc:270] Start()
0924 12:47:12.819333 (+793148us) write_op.cc:276] Timestamp: P: 1790254032026134 usec, L: 0
0924 12:47:12.819339 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:47:12.819563 (+ 224us) log.cc:842] Serialized 354 byte log entry
0924 12:47:12.970161 (+150598us) op_driver.cc:444] REPLICATION: finished
0924 12:47:12.970314 (+ 153us) write_op.cc:301] APPLY: starting
0924 12:47:12.970335 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:12.970374 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:12.970408 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:12.970501 (+ 93us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:12.970531 (+ 30us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:12.970536 (+ 5us) write_op.cc:312] APPLY: finished
0924 12:47:12.970635 (+ 99us) log.cc:842] Serialized 22 byte log entry
0924 12:47:12.970720 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:12.970737 (+ 17us) write_op.cc:454] Released schema lock
0924 12:47:12.970743 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":82,"num_ops":1,"prepare.queue_time_us":461536,"prepare.run_cpu_time_us":776,"prepare.run_wall_time_us":793842,"replication_time_us":150784}]]}
I20260924 12:47:12.984831 8246 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:12.985028 8246 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:12.985754 8246 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:12.986361 8138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:12.986788 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:12.987289 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
W20260924 12:47:13.259272 7836 tablet_service.cc:2970] error setting up scanner a097f304a130446bade7c8b65aa004b6 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.272s (maximum allowed 0.100s): new_scan_request { tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\357\271\324\325\006\"3\n\005slave\022*\n f7080e4ffff44a22a72b8608c75dca3e\020\001\030\001 \001(\001" signature: "<redacted>""\225\227\301p\373\026\334\232\225\005f\212x\357\265\2618\021`0\217\300\347\n\235Q\\t\253cU\353d\3312a\337\313\342\335X\221\266\t\221\034\225jV\301,:\201\203\335\022)\235}P\271\374\270\330" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b1e701df8a3f4c2b8c2f0ef68abb2436"
I20260924 12:47:13.804162 8319 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:13.804349 8319 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:13.805186 8319 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:13.805694 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:13.805720 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:13.806299 8073 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:13.806852 8319 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:13.841944 8340 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=20, attempt_no=0}) took 1136 ms. Trace:
I20260924 12:47:13.842111 8340 rpcz_store.cc:276] 0924 12:47:12.705115 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:12.705194 (+ 79us) service_pool.cc:227] Handling call
0924 12:47:13.841873 (+1136679us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:12.819694 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:12.819821 (+ 127us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:12.819827 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:47:12.819829 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:12.819931 (+ 102us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:12.819944 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:12.819949 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:12.819997 (+ 48us) tablet.cc:699] Row locks acquired
0924 12:47:12.819998 (+ 1us) write_op.cc:260] PREPARE: finished
0924 12:47:12.820038 (+ 40us) write_op.cc:270] Start()
0924 12:47:12.951184 (+131146us) write_op.cc:276] Timestamp: P: 1790254032820005 usec, L: 0
0924 12:47:12.951190 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:47:12.951399 (+ 209us) log.cc:842] Serialized 354 byte log entry
0924 12:47:13.840851 (+889452us) op_driver.cc:444] REPLICATION: finished
0924 12:47:13.841356 (+ 505us) write_op.cc:301] APPLY: starting
0924 12:47:13.841375 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:13.841412 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:13.841438 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:13.841519 (+ 81us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:13.841546 (+ 27us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:13.841550 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:47:13.841652 (+ 102us) log.cc:842] Serialized 22 byte log entry
0924 12:47:13.841732 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:13.841753 (+ 21us) write_op.cc:454] Released schema lock
0924 12:47:13.841761 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":436,"num_ops":1,"prepare.queue_time_us":113997,"prepare.run_cpu_time_us":718,"prepare.run_wall_time_us":131770,"replication_time_us":889625,"thread_start_us":294,"threads_started":1}]]}
I20260924 12:47:14.014984 8246 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:14.016114 7876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.5.77.194:39441 (request call id 2841) took 1029 ms. Trace:
I20260924 12:47:14.016215 7876 rpcz_store.cc:276] 0924 12:47:12.986548 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:12.986685 (+ 137us) service_pool.cc:227] Handling call
0924 12:47:12.986878 (+ 193us) raft_consensus.cc:1457] Updating replica for 2 ops
0924 12:47:12.987017 (+ 139us) raft_consensus.cc:1520] Early marking committed up to index 20
0924 12:47:12.987397 (+ 380us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0924 12:47:12.987541 (+ 144us) write_op.cc:270] Start()
0924 12:47:13.223709 (+236168us) write_op.cc:276] Timestamp: P: 1790254032026134 usec, L: 0
0924 12:47:13.224044 (+ 335us) write_op.cc:270] Start()
0924 12:47:14.013245 (+789201us) write_op.cc:276] Timestamp: P: 1790254032820005 usec, L: 0
0924 12:47:14.013968 (+ 723us) log.cc:842] Serialized 708 byte log entry
0924 12:47:14.014311 (+ 343us) raft_consensus.cc:1631] Marking committed up to 21
0924 12:47:14.014868 (+ 557us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 12:47:14.014906 (+ 38us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 12:47:14.016004 (+ 1098us) raft_consensus.cc:1671] finished
0924 12:47:14.016006 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 12:47:14.016061 (+ 55us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:13.223952 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:13.224060 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:13.224067 (+ 7us) write_op.cc:435] Acquired schema lock
0924 12:47:13.224069 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:13.224161 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:13.224174 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:13.224177 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:13.224226 (+ 49us) tablet.cc:699] Row locks acquired
0924 12:47:13.224227 (+ 1us) write_op.cc:260] PREPARE: finished
0924 12:47:14.014359 (+790132us) op_driver.cc:444] REPLICATION: finished
0924 12:47:14.014931 (+ 572us) write_op.cc:301] APPLY: starting
0924 12:47:14.014947 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:14.014990 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:14.014992 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:14.015069 (+ 77us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:14.015102 (+ 33us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:14.015142 (+ 40us) write_op.cc:312] APPLY: finished
0924 12:47:14.015310 (+ 168us) log.cc:842] Serialized 22 byte log entry
0924 12:47:14.015357 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:14.015371 (+ 14us) write_op.cc:454] Released schema lock
0924 12:47:14.015377 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0924 12:47:14.015437 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:14.015439 (+ 2us) write_op.cc:454] Released schema lock
Related trace 'op':
0924 12:47:14.013789 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:14.013923 (+ 134us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:14.013930 (+ 7us) write_op.cc:435] Acquired schema lock
0924 12:47:14.013932 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:14.014065 (+ 133us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:14.014080 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:14.014085 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:14.014151 (+ 66us) tablet.cc:699] Row locks acquired
0924 12:47:14.014154 (+ 3us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":246,"threads_started":1,"wal-append.queue_time_us":359,"child_traces":[["op",{"apply.queue_time_us":520,"apply.run_cpu_time_us":499,"apply.run_wall_time_us":559,"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":78,"prepare.run_cpu_time_us":370,"prepare.run_wall_time_us":370,"replication_time_us":1026799,"thread_start_us":430,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":405,"prepare.run_cpu_time_us":478,"prepare.run_wall_time_us":478,"thread_start_us":300,"threads_started":1}]]}
I20260924 12:47:14.486634 8319 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:14.486866 8319 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:14.487689 8319 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:14.488067 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:14.488322 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:14.488931 8073 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:14.508881 8347 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=21, attempt_no=0}) took 1536 ms. Trace:
I20260924 12:47:14.509016 8347 rpcz_store.cc:276] 0924 12:47:12.972641 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:12.972705 (+ 64us) service_pool.cc:227] Handling call
0924 12:47:14.508830 (+1536125us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:12.973184 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:12.973276 (+ 92us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:12.973280 (+ 4us) write_op.cc:435] Acquired schema lock
0924 12:47:12.973281 (+ 1us) tablet.cc:660] Decoding operations
0924 12:47:12.973348 (+ 67us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:12.973357 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:12.973360 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:12.973399 (+ 39us) tablet.cc:699] Row locks acquired
0924 12:47:12.973400 (+ 1us) write_op.cc:260] PREPARE: finished
0924 12:47:12.973434 (+ 34us) write_op.cc:270] Start()
0924 12:47:13.812588 (+839154us) write_op.cc:276] Timestamp: P: 1790254032973389 usec, L: 0
0924 12:47:13.812593 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:47:13.812765 (+ 172us) log.cc:842] Serialized 354 byte log entry
0924 12:47:14.507953 (+695188us) op_driver.cc:444] REPLICATION: finished
0924 12:47:14.508383 (+ 430us) write_op.cc:301] APPLY: starting
0924 12:47:14.508400 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:14.508435 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:14.508459 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:14.508533 (+ 74us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:14.508561 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:14.508565 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:47:14.508647 (+ 82us) log.cc:842] Serialized 22 byte log entry
0924 12:47:14.508720 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:14.508736 (+ 16us) write_op.cc:454] Released schema lock
0924 12:47:14.508742 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":367,"num_ops":1,"prepare.queue_time_us":45,"prepare.run_cpu_time_us":589,"prepare.run_wall_time_us":839644,"replication_time_us":695326,"thread_start_us":250,"threads_started":1}]]}
W20260924 12:47:14.526638 7836 tablet_service.cc:2970] error setting up scanner 7214f0117ff4494a9a7b91a4eaceb5fc with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.508s (maximum allowed 0.100s): new_scan_request { tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\357\271\324\325\006\"3\n\005slave\022*\n f7080e4ffff44a22a72b8608c75dca3e\020\001\030\001 \001(\001" signature: "<redacted>""\225\227\301p\373\026\334\232\225\005f\212x\357\265\2618\021`0\217\300\347\n\235Q\\t\253cU\353d\3312a\337\313\342\335X\221\266\t\221\034\225jV\301,:\201\203\335\022)\235}P\271\374\270\330" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "dcfcbbc642ea47f3b43d5c78efc7a745"
I20260924 12:47:14.572573 8246 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:14.572767 8246 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:14.573503 8246 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:14.574030 8138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:14.574398 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:14.575150 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:14.799909 8319 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:14.825039 8246 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:14.986347 8246 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:14.986574 8246 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:14.986902 8319 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:14.987051 8319 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:14.987435 8246 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:14.987810 8319 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:14.988305 8137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:14.988425 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:14.988878 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:14.989388 8073 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:14.989482 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:14.989816 8319 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:14.990067 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:14.990324 8246 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:15.017966 8319 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:15.018150 8319 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:15.018826 8319 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:15.019335 7876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:15.020037 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:15.020655 8074 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:15.021010 8319 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:16.411520 8319 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:16.411703 8319 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:16.412377 8319 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:16.412923 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:16.412947 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:16.413511 8073 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:16.413771 8319 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:16.431221 8351 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:16.431413 8351 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:16.432206 8351 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:16.432684 8138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:16.432953 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:16.433523 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:16.438642 8360 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=23, attempt_no=0}) took 1927 ms. Trace:
I20260924 12:47:16.438747 8360 rpcz_store.cc:276] 0924 12:47:14.510731 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:14.510829 (+ 98us) service_pool.cc:227] Handling call
0924 12:47:16.438599 (+1927770us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:14.511296 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:14.511400 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:14.511405 (+ 5us) write_op.cc:435] Acquired schema lock
0924 12:47:14.511407 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:14.511505 (+ 98us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:14.511516 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:14.511521 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:14.511567 (+ 46us) tablet.cc:699] Row locks acquired
0924 12:47:14.511569 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:14.511603 (+ 34us) write_op.cc:270] Start()
0924 12:47:15.505749 (+994146us) write_op.cc:276] Timestamp: P: 1790254034511564 usec, L: 0
0924 12:47:15.505755 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:47:15.505987 (+ 232us) log.cc:842] Serialized 354 byte log entry
0924 12:47:16.437725 (+931738us) op_driver.cc:444] REPLICATION: finished
0924 12:47:16.438109 (+ 384us) write_op.cc:301] APPLY: starting
0924 12:47:16.438123 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:16.438155 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:16.438181 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:16.438250 (+ 69us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:16.438275 (+ 25us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:16.438279 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:47:16.438375 (+ 96us) log.cc:842] Serialized 22 byte log entry
0924 12:47:16.438489 (+ 114us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:16.438506 (+ 17us) write_op.cc:454] Released schema lock
0924 12:47:16.438513 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":333,"mutex_wait_us":20,"num_ops":1,"prepare.queue_time_us":43,"prepare.run_cpu_time_us":741,"prepare.run_wall_time_us":994813,"raft.queue_time_us":192,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":931945,"thread_start_us":241,"threads_started":1}]]}
I20260924 12:47:16.520634 8319 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:16.520838 8319 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:16.521593 8319 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:16.522109 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:16.522194 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:16.522891 8074 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:16.523185 8319 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:16.567292 8360 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=24, attempt_no=0}) took 1728 ms. Trace:
I20260924 12:47:16.567481 8360 rpcz_store.cc:276] 0924 12:47:14.839061 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:14.839104 (+ 43us) service_pool.cc:227] Handling call
0924 12:47:16.567228 (+1728124us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:15.506157 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:15.506301 (+ 144us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:15.506307 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:47:15.506310 (+ 3us) tablet.cc:660] Decoding operations
0924 12:47:15.506407 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:15.506419 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:15.506424 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:15.506500 (+ 76us) tablet.cc:699] Row locks acquired
0924 12:47:15.506503 (+ 3us) write_op.cc:260] PREPARE: finished
0924 12:47:15.506552 (+ 49us) write_op.cc:270] Start()
0924 12:47:15.529699 (+ 23147us) write_op.cc:276] Timestamp: P: 1790254035506511 usec, L: 0
0924 12:47:15.529703 (+ 4us) op_driver.cc:331] REPLICATION: starting
0924 12:47:15.529912 (+ 209us) log.cc:842] Serialized 354 byte log entry
0924 12:47:16.566665 (+1036753us) op_driver.cc:444] REPLICATION: finished
0924 12:47:16.566776 (+ 111us) write_op.cc:301] APPLY: starting
0924 12:47:16.566791 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:16.566826 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:16.566853 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:16.566914 (+ 61us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:16.566943 (+ 29us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:16.566947 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:47:16.567020 (+ 73us) log.cc:842] Serialized 22 byte log entry
0924 12:47:16.567094 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:16.567113 (+ 19us) write_op.cc:454] Released schema lock
0924 12:47:16.567120 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":62,"num_ops":1,"prepare.queue_time_us":666581,"prepare.run_cpu_time_us":774,"prepare.run_wall_time_us":23842,"replication_time_us":1036928}]]}
I20260924 12:47:16.602399 8351 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:16.988142 8319 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:16.988327 8319 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:16.988986 8319 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:16.989317 7876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:16.989534 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:16.990049 8074 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:16.990306 8319 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 12:47:17.345515 7836 tablet_service.cc:2970] error setting up scanner 908e7eb076064ae4a93570b4ef7536c7 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.742s (maximum allowed 0.100s): new_scan_request { tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\357\271\324\325\006\"3\n\005slave\022*\n f7080e4ffff44a22a72b8608c75dca3e\020\001\030\001 \001(\001" signature: "<redacted>""\225\227\301p\373\026\334\232\225\005f\212x\357\265\2618\021`0\217\300\347\n\235Q\\t\253cU\353d\3312a\337\313\342\335X\221\266\t\221\034\225jV\301,:\201\203\335\022)\235}P\271\374\270\330" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "dcfcbbc642ea47f3b43d5c78efc7a745"
I20260924 12:47:17.741695 8363 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:17.741904 8363 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:17.742794 8363 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:17.743362 8138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:17.743539 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:17.744182 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:17.757757 8319 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:17.757953 8319 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:17.758755 8319 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:17.759195 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:17.759223 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:17.759900 8074 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:17.775113 8370 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=25, attempt_no=0}) took 1334 ms. Trace:
I20260924 12:47:17.775264 8370 rpcz_store.cc:276] 0924 12:47:16.440398 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:16.440570 (+ 172us) service_pool.cc:227] Handling call
0924 12:47:17.775055 (+1334485us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:16.441312 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:16.441437 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:16.441441 (+ 4us) write_op.cc:435] Acquired schema lock
0924 12:47:16.441442 (+ 1us) tablet.cc:660] Decoding operations
0924 12:47:16.441557 (+ 115us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:16.441566 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:16.441569 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:16.441617 (+ 48us) tablet.cc:699] Row locks acquired
0924 12:47:16.441619 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:16.441652 (+ 33us) write_op.cc:270] Start()
0924 12:47:17.242807 (+801155us) write_op.cc:276] Timestamp: P: 1790254036441630 usec, L: 0
0924 12:47:17.242812 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:47:17.243073 (+ 261us) log.cc:842] Serialized 354 byte log entry
0924 12:47:17.773982 (+530909us) op_driver.cc:444] REPLICATION: finished
0924 12:47:17.774415 (+ 433us) write_op.cc:301] APPLY: starting
0924 12:47:17.774436 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:17.774506 (+ 70us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:17.774532 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:17.774604 (+ 72us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:17.774639 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:17.774644 (+ 5us) write_op.cc:312] APPLY: finished
0924 12:47:17.774792 (+ 148us) log.cc:842] Serialized 22 byte log entry
0924 12:47:17.774874 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:17.774896 (+ 22us) write_op.cc:454] Released schema lock
0924 12:47:17.774904 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":353,"num_ops":1,"prepare.queue_time_us":332,"prepare.run_cpu_time_us":779,"prepare.run_wall_time_us":801836,"replication_time_us":531130,"thread_start_us":450,"threads_started":2}]]}
W20260924 12:47:17.870175 7836 tablet_service.cc:2970] error setting up scanner d4eb6376b40f40faa8f027ee11f6b351 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.126s (maximum allowed 0.100s): new_scan_request { tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\357\271\324\325\006\"3\n\005slave\022*\n f7080e4ffff44a22a72b8608c75dca3e\020\001\030\001 \001(\001" signature: "<redacted>""\225\227\301p\373\026\334\232\225\005f\212x\357\265\2618\021`0\217\300\347\n\235Q\\t\253cU\353d\3312a\337\313\342\335X\221\266\t\221\034\225jV\301,:\201\203\335\022)\235}P\271\374\270\330" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b1e701df8a3f4c2b8c2f0ef68abb2436"
I20260924 12:47:18.040954 8319 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:18.057668 8370 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=26, attempt_no=0}) took 1488 ms. Trace:
I20260924 12:47:18.057806 8370 rpcz_store.cc:276] 0924 12:47:16.569470 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:16.569541 (+ 71us) service_pool.cc:227] Handling call
0924 12:47:18.057600 (+1488059us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:17.243195 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:17.243313 (+ 118us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:17.243319 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:47:17.243321 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:17.243412 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:17.243423 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:17.243428 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:17.243479 (+ 51us) tablet.cc:699] Row locks acquired
0924 12:47:17.243482 (+ 3us) write_op.cc:260] PREPARE: finished
0924 12:47:17.243531 (+ 49us) write_op.cc:270] Start()
0924 12:47:17.461706 (+218175us) write_op.cc:276] Timestamp: P: 1790254037243468 usec, L: 0
0924 12:47:17.461712 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:47:17.461979 (+ 267us) log.cc:842] Serialized 354 byte log entry
0924 12:47:18.057038 (+595059us) op_driver.cc:444] REPLICATION: finished
0924 12:47:18.057132 (+ 94us) write_op.cc:301] APPLY: starting
0924 12:47:18.057151 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:18.057190 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:18.057219 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:18.057282 (+ 63us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:18.057311 (+ 29us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:18.057315 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:47:18.057387 (+ 72us) log.cc:842] Serialized 22 byte log entry
0924 12:47:18.057479 (+ 92us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:18.057498 (+ 19us) write_op.cc:454] Released schema lock
0924 12:47:18.057505 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":34,"num_ops":1,"prepare.queue_time_us":673181,"prepare.run_cpu_time_us":793,"prepare.run_wall_time_us":218870,"replication_time_us":595272}]]}
I20260924 12:47:18.603374 8363 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:19.226327 8319 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:19.226572 8319 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:19.227281 8319 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:19.227748 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:19.227805 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:19.228514 8073 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:19.228783 8319 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:19.232434 8378 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=27, attempt_no=0}) took 1455 ms. Trace:
I20260924 12:47:19.232539 8378 rpcz_store.cc:276] 0924 12:47:17.776917 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:17.777006 (+ 89us) service_pool.cc:227] Handling call
0924 12:47:19.232382 (+1455376us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:17.777531 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:17.777624 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:17.777628 (+ 4us) write_op.cc:435] Acquired schema lock
0924 12:47:17.777630 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:17.777709 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:17.777718 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:17.777721 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:17.777762 (+ 41us) tablet.cc:699] Row locks acquired
0924 12:47:17.777763 (+ 1us) write_op.cc:260] PREPARE: finished
0924 12:47:17.777796 (+ 33us) write_op.cc:270] Start()
0924 12:47:18.274947 (+497151us) write_op.cc:276] Timestamp: P: 1790254037777731 usec, L: 0
0924 12:47:18.274953 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:47:18.275210 (+ 257us) log.cc:842] Serialized 354 byte log entry
0924 12:47:19.231445 (+956235us) op_driver.cc:444] REPLICATION: finished
0924 12:47:19.231872 (+ 427us) write_op.cc:301] APPLY: starting
0924 12:47:19.231887 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:19.231919 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:19.231962 (+ 43us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:19.232073 (+ 111us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:19.232101 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:19.232105 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:47:19.232177 (+ 72us) log.cc:842] Serialized 22 byte log entry
0924 12:47:19.232271 (+ 94us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:19.232285 (+ 14us) write_op.cc:454] Released schema lock
0924 12:47:19.232291 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":375,"num_ops":1,"prepare.queue_time_us":41,"prepare.run_cpu_time_us":684,"prepare.run_wall_time_us":497751,"replication_time_us":956468,"spinlock_wait_cycles":22272,"thread_start_us":282,"threads_started":1}]]}
I20260924 12:47:19.265327 8319 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:19.265529 8319 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:19.266235 8319 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:19.266803 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:19.266819 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:19.267424 8074 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:19.267803 8319 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:19.306430 8378 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=28, attempt_no=0}) took 1246 ms. Trace:
I20260924 12:47:19.306607 8378 rpcz_store.cc:276] 0924 12:47:18.059526 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:18.059592 (+ 66us) service_pool.cc:227] Handling call
0924 12:47:19.306368 (+1246776us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:18.275323 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:18.275465 (+ 142us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:18.275472 (+ 7us) write_op.cc:435] Acquired schema lock
0924 12:47:18.275475 (+ 3us) tablet.cc:660] Decoding operations
0924 12:47:18.275579 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:18.275592 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:18.275597 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:18.275652 (+ 55us) tablet.cc:699] Row locks acquired
0924 12:47:18.275654 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:18.275691 (+ 37us) write_op.cc:270] Start()
0924 12:47:18.624831 (+349140us) write_op.cc:276] Timestamp: P: 1790254038275650 usec, L: 0
0924 12:47:18.624836 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:47:18.625075 (+ 239us) log.cc:842] Serialized 354 byte log entry
0924 12:47:19.305710 (+680635us) op_driver.cc:444] REPLICATION: finished
0924 12:47:19.305810 (+ 100us) write_op.cc:301] APPLY: starting
0924 12:47:19.305830 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:19.305866 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:19.305894 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:19.305986 (+ 92us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:19.306024 (+ 38us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:19.306029 (+ 5us) write_op.cc:312] APPLY: finished
0924 12:47:19.306129 (+ 100us) log.cc:842] Serialized 22 byte log entry
0924 12:47:19.306238 (+ 109us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:19.306259 (+ 21us) write_op.cc:454] Released schema lock
0924 12:47:19.306268 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":43,"num_ops":1,"prepare.queue_time_us":215350,"prepare.run_cpu_time_us":764,"prepare.run_wall_time_us":349823,"replication_time_us":680840}]]}
I20260924 12:47:19.341498 8363 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:19.341686 8363 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:19.342377 8363 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:19.343153 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:19.343523 8137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:19.344192 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:19.372756 8319 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:19.372924 8319 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:19.373596 8319 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:19.374135 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:19.374650 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:19.375115 8073 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:19.375353 8319 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:20.234639 8319 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:20.234838 8319 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:20.235598 8319 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:20.236047 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:20.236069 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:20.236703 8073 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:20.237102 8319 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:20.269399 8363 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:20.280999 8388 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=29, attempt_no=0}) took 1046 ms. Trace:
I20260924 12:47:20.281246 8388 rpcz_store.cc:276] 0924 12:47:19.234277 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:19.234351 (+ 74us) service_pool.cc:227] Handling call
0924 12:47:20.280947 (+1046596us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:19.235106 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:19.235256 (+ 150us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:19.235260 (+ 4us) write_op.cc:435] Acquired schema lock
0924 12:47:19.235262 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:19.235348 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:19.235357 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:19.235360 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:19.235413 (+ 53us) tablet.cc:699] Row locks acquired
0924 12:47:19.235415 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:19.235457 (+ 42us) write_op.cc:270] Start()
0924 12:47:19.826582 (+591125us) write_op.cc:276] Timestamp: P: 1790254039235401 usec, L: 0
0924 12:47:19.826588 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:47:19.826791 (+ 203us) log.cc:842] Serialized 354 byte log entry
0924 12:47:20.279958 (+453167us) op_driver.cc:444] REPLICATION: finished
0924 12:47:20.280452 (+ 494us) write_op.cc:301] APPLY: starting
0924 12:47:20.280467 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:20.280505 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:20.280524 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:20.280615 (+ 91us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:20.280643 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:20.280647 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:47:20.280730 (+ 83us) log.cc:842] Serialized 22 byte log entry
0924 12:47:20.280809 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:20.280827 (+ 18us) write_op.cc:454] Released schema lock
0924 12:47:20.280834 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":441,"num_ops":1,"prepare.queue_time_us":314,"prepare.run_cpu_time_us":1840,"prepare.run_wall_time_us":592398,"raft.queue_time_us":204,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":453343,"thread_start_us":1128,"threads_started":3}]]}
I20260924 12:47:20.748133 8319 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:20.748343 8319 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:20.749120 8319 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:20.749544 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:20.749810 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:20.750278 8073 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:20.779927 8388 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=30, attempt_no=0}) took 1471 ms. Trace:
I20260924 12:47:20.780049 8388 rpcz_store.cc:276] 0924 12:47:19.308255 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:19.308313 (+ 58us) service_pool.cc:227] Handling call
0924 12:47:20.779865 (+1471552us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:19.827520 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:19.827646 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:19.827653 (+ 7us) write_op.cc:435] Acquired schema lock
0924 12:47:19.827655 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:19.827747 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:19.827759 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:19.827763 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:19.827818 (+ 55us) tablet.cc:699] Row locks acquired
0924 12:47:19.827820 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:19.827863 (+ 43us) write_op.cc:270] Start()
0924 12:47:20.633013 (+805150us) write_op.cc:276] Timestamp: P: 1790254039827759 usec, L: 0
0924 12:47:20.633019 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:47:20.633231 (+ 212us) log.cc:842] Serialized 354 byte log entry
0924 12:47:20.779222 (+145991us) op_driver.cc:444] REPLICATION: finished
0924 12:47:20.779338 (+ 116us) write_op.cc:301] APPLY: starting
0924 12:47:20.779357 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:20.779393 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:20.779422 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:20.779523 (+ 101us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:20.779554 (+ 31us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:20.779558 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:47:20.779655 (+ 97us) log.cc:842] Serialized 22 byte log entry
0924 12:47:20.779746 (+ 91us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:20.779766 (+ 20us) write_op.cc:454] Released schema lock
0924 12:47:20.779772 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":54,"num_ops":1,"prepare.queue_time_us":518797,"prepare.run_cpu_time_us":711,"prepare.run_wall_time_us":805783,"replication_time_us":146166}]]}
I20260924 12:47:20.954665 8363 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:20.954872 8363 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:20.955703 8363 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:20.956512 8138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:20.956569 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:20.957233 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:21.543087 8319 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:21.556519 8398 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=31, attempt_no=0}) took 1273 ms. Trace:
I20260924 12:47:21.556648 8398 rpcz_store.cc:276] 0924 12:47:20.283108 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:20.283168 (+ 60us) service_pool.cc:227] Handling call
0924 12:47:21.556453 (+1273285us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:20.633343 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:20.633466 (+ 123us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:20.633473 (+ 7us) write_op.cc:435] Acquired schema lock
0924 12:47:20.633475 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:20.633568 (+ 93us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:20.633580 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:20.633585 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:20.633641 (+ 56us) tablet.cc:699] Row locks acquired
0924 12:47:20.633643 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:20.633692 (+ 49us) write_op.cc:270] Start()
0924 12:47:20.712872 (+ 79180us) write_op.cc:276] Timestamp: P: 1790254040633656 usec, L: 0
0924 12:47:20.712878 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:47:20.713107 (+ 229us) log.cc:842] Serialized 354 byte log entry
0924 12:47:21.555469 (+842362us) op_driver.cc:444] REPLICATION: finished
0924 12:47:21.555940 (+ 471us) write_op.cc:301] APPLY: starting
0924 12:47:21.555958 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:21.555992 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:21.556016 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:21.556126 (+ 110us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:21.556153 (+ 27us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:21.556158 (+ 5us) write_op.cc:312] APPLY: finished
0924 12:47:21.556259 (+ 101us) log.cc:842] Serialized 22 byte log entry
0924 12:47:21.556336 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:21.556353 (+ 17us) write_op.cc:454] Released schema lock
0924 12:47:21.556359 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":389,"num_ops":1,"prepare.queue_time_us":349646,"prepare.run_cpu_time_us":769,"prepare.run_wall_time_us":79849,"replication_time_us":842551,"thread_start_us":264,"threads_started":1}]]}
I20260924 12:47:21.626269 8319 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:21.626535 8319 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:21.627326 8319 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:21.627741 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:21.627986 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:21.628561 8073 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:21.628815 8319 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:21.997262 8363 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:21.997511 7876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.5.77.194:39441 (request call id 3806) took 1041 ms. Trace:
I20260924 12:47:21.997656 7876 rpcz_store.cc:276] 0924 12:47:20.956358 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:20.956422 (+ 64us) service_pool.cc:227] Handling call
0924 12:47:20.956680 (+ 258us) raft_consensus.cc:1457] Updating replica for 2 ops
0924 12:47:20.956801 (+ 121us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 12:47:20.956805 (+ 4us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0924 12:47:20.956933 (+ 128us) write_op.cc:270] Start()
0924 12:47:21.542113 (+585180us) write_op.cc:276] Timestamp: P: 1790254039827759 usec, L: 0
0924 12:47:21.542943 (+ 830us) write_op.cc:270] Start()
0924 12:47:21.996102 (+453159us) write_op.cc:276] Timestamp: P: 1790254040633656 usec, L: 0
0924 12:47:21.996508 (+ 406us) log.cc:842] Serialized 708 byte log entry
0924 12:47:21.996828 (+ 320us) raft_consensus.cc:1631] Marking committed up to 32
0924 12:47:21.997128 (+ 300us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 12:47:21.997168 (+ 40us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 12:47:21.997386 (+ 218us) raft_consensus.cc:1671] finished
0924 12:47:21.997388 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 12:47:21.997436 (+ 48us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:21.542828 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:21.542932 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:21.542937 (+ 5us) write_op.cc:435] Acquired schema lock
0924 12:47:21.542939 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:21.543031 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:21.543043 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:21.543046 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:21.543092 (+ 46us) tablet.cc:699] Row locks acquired
0924 12:47:21.543094 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:21.996860 (+453766us) op_driver.cc:444] REPLICATION: finished
0924 12:47:21.997253 (+ 393us) write_op.cc:301] APPLY: starting
0924 12:47:21.997269 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:21.997317 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:21.997320 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:21.997391 (+ 71us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:21.997424 (+ 33us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:21.997451 (+ 27us) write_op.cc:312] APPLY: finished
0924 12:47:21.997575 (+ 124us) log.cc:842] Serialized 22 byte log entry
0924 12:47:21.997647 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:21.997664 (+ 17us) write_op.cc:454] Released schema lock
0924 12:47:21.997671 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0924 12:47:21.997759 (+ 88us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:21.997763 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0924 12:47:21.996318 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:21.996448 (+ 130us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:21.996456 (+ 8us) write_op.cc:435] Acquired schema lock
0924 12:47:21.996458 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:21.996557 (+ 99us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:21.996568 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:21.996572 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:21.996619 (+ 47us) tablet.cc:699] Row locks acquired
0924 12:47:21.996620 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":231,"threads_started":1,"wal-append.queue_time_us":324,"child_traces":[["op",{"apply.queue_time_us":333,"apply.run_cpu_time_us":575,"apply.run_wall_time_us":575,"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":595,"prepare.run_cpu_time_us":370,"prepare.run_wall_time_us":370,"replication_time_us":1039915,"thread_start_us":710,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":61,"prepare.run_cpu_time_us":403,"prepare.run_wall_time_us":402}]]}
W20260924 12:47:22.212399 7836 tablet_service.cc:2970] error setting up scanner 98dd97562a3a44b7be30b479bb7b65e0 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.213s (maximum allowed 0.100s): new_scan_request { tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\357\271\324\325\006\"3\n\005slave\022*\n f7080e4ffff44a22a72b8608c75dca3e\020\001\030\001 \001(\001" signature: "<redacted>""\225\227\301p\373\026\334\232\225\005f\212x\357\265\2618\021`0\217\300\347\n\235Q\\t\253cU\353d\3312a\337\313\342\335X\221\266\t\221\034\225jV\301,:\201\203\335\022)\235}P\271\374\270\330" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "dcfcbbc642ea47f3b43d5c78efc7a745"
W20260924 12:47:22.481160 7836 tablet_service.cc:2970] error setting up scanner 53c2eef9dc3d471385ceed8c4b3adaaf with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.482s (maximum allowed 0.100s): new_scan_request { tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\357\271\324\325\006\"3\n\005slave\022*\n f7080e4ffff44a22a72b8608c75dca3e\020\001\030\001 \001(\001" signature: "<redacted>""\225\227\301p\373\026\334\232\225\005f\212x\357\265\2618\021`0\217\300\347\n\235Q\\t\253cU\353d\3312a\337\313\342\335X\221\266\t\221\034\225jV\301,:\201\203\335\022)\235}P\271\374\270\330" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b1e701df8a3f4c2b8c2f0ef68abb2436"
I20260924 12:47:22.669723 8363 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:22.669917 8363 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:22.670722 8363 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:22.671252 8138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:22.671537 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:22.672025 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:23.156502 8363 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:23.176983 8406 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=33, attempt_no=0}) took 1618 ms. Trace:
I20260924 12:47:23.177129 8406 rpcz_store.cc:276] 0924 12:47:21.558383 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:21.558472 (+ 89us) service_pool.cc:227] Handling call
0924 12:47:23.176926 (+1618454us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:21.559338 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:21.559473 (+ 135us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:21.559478 (+ 5us) write_op.cc:435] Acquired schema lock
0924 12:47:21.559479 (+ 1us) tablet.cc:660] Decoding operations
0924 12:47:21.559550 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:21.559560 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:21.559563 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:21.559609 (+ 46us) tablet.cc:699] Row locks acquired
0924 12:47:21.559611 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:21.559644 (+ 33us) write_op.cc:270] Start()
0924 12:47:22.394796 (+835152us) write_op.cc:276] Timestamp: P: 1790254041559616 usec, L: 0
0924 12:47:22.394801 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:47:22.395085 (+ 284us) log.cc:842] Serialized 354 byte log entry
0924 12:47:23.175876 (+780791us) op_driver.cc:444] REPLICATION: finished
0924 12:47:23.176318 (+ 442us) write_op.cc:301] APPLY: starting
0924 12:47:23.176336 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:23.176370 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:23.176400 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:23.176499 (+ 99us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:23.176527 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:23.176531 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:47:23.176665 (+ 134us) log.cc:842] Serialized 22 byte log entry
0924 12:47:23.176738 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:23.176779 (+ 41us) write_op.cc:454] Released schema lock
0924 12:47:23.176789 (+ 10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":379,"num_ops":1,"prepare.queue_time_us":333,"prepare.run_cpu_time_us":845,"prepare.run_wall_time_us":835908,"raft.queue_time_us":206,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":781039,"spinlock_wait_cycles":56704,"thread_start_us":468,"threads_started":2}]]}
I20260924 12:47:23.299458 8402 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:23.299680 8402 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:23.300478 8402 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:23.300911 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:23.301025 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:23.301565 8073 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:23.676802 8402 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:23.697355 8411 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=34, attempt_no=0}) took 2050 ms. Trace:
I20260924 12:47:23.697590 8411 rpcz_store.cc:276] 0924 12:47:21.646448 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:21.646526 (+ 78us) service_pool.cc:227] Handling call
0924 12:47:23.697280 (+2050754us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:22.395279 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:22.395396 (+ 117us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:22.395403 (+ 7us) write_op.cc:435] Acquired schema lock
0924 12:47:22.395405 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:22.395497 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:22.395508 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:22.395514 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:22.395562 (+ 48us) tablet.cc:699] Row locks acquired
0924 12:47:22.395563 (+ 1us) write_op.cc:260] PREPARE: finished
0924 12:47:22.395607 (+ 44us) write_op.cc:270] Start()
0924 12:47:22.754757 (+359150us) write_op.cc:276] Timestamp: P: 1790254042395549 usec, L: 0
0924 12:47:22.754763 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:47:22.754987 (+ 224us) log.cc:842] Serialized 354 byte log entry
0924 12:47:23.696115 (+941128us) op_driver.cc:444] REPLICATION: finished
0924 12:47:23.696748 (+ 633us) write_op.cc:301] APPLY: starting
0924 12:47:23.696777 (+ 29us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:23.696822 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:23.696854 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:23.696930 (+ 76us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:23.696960 (+ 30us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:23.696964 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:47:23.697069 (+ 105us) log.cc:842] Serialized 22 byte log entry
0924 12:47:23.697156 (+ 87us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:23.697176 (+ 20us) write_op.cc:454] Released schema lock
0924 12:47:23.697184 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":568,"num_ops":1,"prepare.queue_time_us":748272,"prepare.run_cpu_time_us":727,"prepare.run_wall_time_us":359787,"replication_time_us":941319,"thread_start_us":445,"threads_started":1}]]}
I20260924 12:47:23.896955 8363 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:23.897141 8363 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:23.897992 8363 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:23.898190 8137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:23.898574 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:23.899133 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:23.899447 8363 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:24.016242 8402 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:24.016464 8402 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:24.017036 8402 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:24.017398 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:24.017519 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:24.018107 8073 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:24.018363 8402 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:24.762435 8402 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:24.762660 8402 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:24.763774 8402 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:24.764385 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:24.764467 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:24.765120 8073 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:24.765555 8402 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:24.769320 8419 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=35, attempt_no=0}) took 1590 ms. Trace:
I20260924 12:47:24.769456 8419 rpcz_store.cc:276] 0924 12:47:23.179045 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:23.179130 (+ 85us) service_pool.cc:227] Handling call
0924 12:47:24.769267 (+1590137us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:23.179660 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:23.179787 (+ 127us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:23.179792 (+ 5us) write_op.cc:435] Acquired schema lock
0924 12:47:23.179793 (+ 1us) tablet.cc:660] Decoding operations
0924 12:47:23.179867 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:23.179876 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:23.179879 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:23.179922 (+ 43us) tablet.cc:699] Row locks acquired
0924 12:47:23.179924 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:23.179959 (+ 35us) write_op.cc:270] Start()
0924 12:47:24.013112 (+833153us) write_op.cc:276] Timestamp: P: 1790254043179920 usec, L: 0
0924 12:47:24.013118 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:47:24.013351 (+ 233us) log.cc:842] Serialized 354 byte log entry
0924 12:47:24.768152 (+754801us) op_driver.cc:444] REPLICATION: finished
0924 12:47:24.768736 (+ 584us) write_op.cc:301] APPLY: starting
0924 12:47:24.768753 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:24.768789 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:24.768855 (+ 66us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:24.768954 (+ 99us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:24.768981 (+ 27us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:24.768986 (+ 5us) write_op.cc:312] APPLY: finished
0924 12:47:24.769064 (+ 78us) log.cc:842] Serialized 22 byte log entry
0924 12:47:24.769151 (+ 87us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:24.769166 (+ 15us) write_op.cc:454] Released schema lock
0924 12:47:24.769171 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":507,"num_ops":1,"prepare.queue_time_us":42,"prepare.run_cpu_time_us":712,"prepare.run_wall_time_us":833768,"replication_time_us":755002,"thread_start_us":370,"threads_started":1}]]}
I20260924 12:47:24.801028 8363 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:24.801268 8363 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:24.802088 8363 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:24.802704 8138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:24.803053 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:24.803752 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:24.833241 8363 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:24.835134 8419 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=36, attempt_no=0}) took 1135 ms. Trace:
I20260924 12:47:24.835261 8419 rpcz_store.cc:276] 0924 12:47:23.699424 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:23.699472 (+ 48us) service_pool.cc:227] Handling call
0924 12:47:24.835055 (+1135583us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:24.013469 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:24.013593 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:24.013601 (+ 8us) write_op.cc:435] Acquired schema lock
0924 12:47:24.013603 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:24.013703 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:24.013715 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:24.013719 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:24.013782 (+ 63us) tablet.cc:699] Row locks acquired
0924 12:47:24.013784 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:24.013829 (+ 45us) write_op.cc:270] Start()
0924 12:47:24.409980 (+396151us) write_op.cc:276] Timestamp: P: 1790254044013811 usec, L: 0
0924 12:47:24.409985 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:47:24.410207 (+ 222us) log.cc:842] Serialized 354 byte log entry
0924 12:47:24.834434 (+424227us) op_driver.cc:444] REPLICATION: finished
0924 12:47:24.834539 (+ 105us) write_op.cc:301] APPLY: starting
0924 12:47:24.834557 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:24.834609 (+ 52us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:24.834641 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:24.834715 (+ 74us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:24.834752 (+ 37us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:24.834758 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:47:24.834846 (+ 88us) log.cc:842] Serialized 22 byte log entry
0924 12:47:24.834925 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:24.834943 (+ 18us) write_op.cc:454] Released schema lock
0924 12:47:24.834951 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":24,"num_ops":1,"prepare.queue_time_us":313513,"prepare.run_cpu_time_us":765,"prepare.run_wall_time_us":396820,"replication_time_us":424401}]]}
I20260924 12:47:25.305254 8402 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:25.305469 8402 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:25.306272 8402 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:25.306877 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:25.307580 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:25.308108 8074 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:25.781701 8363 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:25.781900 8363 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:25.782693 8363 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:25.783416 8138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:25.783450 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:25.784068 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:25.829288 8427 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=37, attempt_no=0}) took 1057 ms. Trace:
I20260924 12:47:25.829434 8427 rpcz_store.cc:276] 0924 12:47:24.771331 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:24.771406 (+ 75us) service_pool.cc:227] Handling call
0924 12:47:25.829219 (+1057813us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:24.771928 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:24.772061 (+ 133us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:24.772067 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:47:24.772069 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:24.772146 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:24.772155 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:24.772159 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:24.772201 (+ 42us) tablet.cc:699] Row locks acquired
0924 12:47:24.772203 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:24.772237 (+ 34us) write_op.cc:270] Start()
0924 12:47:24.950370 (+178133us) write_op.cc:276] Timestamp: P: 1790254044772194 usec, L: 0
0924 12:47:24.950374 (+ 4us) op_driver.cc:331] REPLICATION: starting
0924 12:47:24.950619 (+ 245us) log.cc:842] Serialized 354 byte log entry
0924 12:47:25.828224 (+877605us) op_driver.cc:444] REPLICATION: finished
0924 12:47:25.828715 (+ 491us) write_op.cc:301] APPLY: starting
0924 12:47:25.828732 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:25.828768 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:25.828793 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:25.828866 (+ 73us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:25.828891 (+ 25us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:25.828895 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:47:25.828987 (+ 92us) log.cc:842] Serialized 22 byte log entry
0924 12:47:25.829073 (+ 86us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:25.829098 (+ 25us) write_op.cc:454] Released schema lock
0924 12:47:25.829106 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":423,"num_ops":1,"prepare.queue_time_us":69,"prepare.run_cpu_time_us":705,"prepare.run_wall_time_us":178770,"replication_time_us":877798,"thread_start_us":253,"threads_started":1}]]}
I20260924 12:47:25.887400 8402 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:25.909297 8402 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:25.909480 8402 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:25.910228 8402 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:25.910658 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:25.910777 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:25.911437 8073 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:25.911764 8402 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:25.913982 8427 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=38, attempt_no=0}) took 1076 ms. Trace:
I20260924 12:47:25.914093 8427 rpcz_store.cc:276] 0924 12:47:24.837133 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:24.837210 (+ 77us) service_pool.cc:227] Handling call
0924 12:47:25.913921 (+1076711us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:24.950739 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:24.950890 (+ 151us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:24.950896 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:47:24.950898 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:24.950997 (+ 99us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:24.951009 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:24.951014 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:24.951070 (+ 56us) tablet.cc:699] Row locks acquired
0924 12:47:24.951072 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:24.951119 (+ 47us) write_op.cc:270] Start()
0924 12:47:25.395260 (+444141us) write_op.cc:276] Timestamp: P: 1790254044951061 usec, L: 0
0924 12:47:25.395266 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:47:25.395490 (+ 224us) log.cc:842] Serialized 354 byte log entry
0924 12:47:25.913452 (+517962us) op_driver.cc:444] REPLICATION: finished
0924 12:47:25.913535 (+ 83us) write_op.cc:301] APPLY: starting
0924 12:47:25.913548 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:25.913583 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:25.913602 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:25.913661 (+ 59us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:25.913687 (+ 26us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:25.913691 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:47:25.913750 (+ 59us) log.cc:842] Serialized 22 byte log entry
0924 12:47:25.913818 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:25.913834 (+ 16us) write_op.cc:454] Released schema lock
0924 12:47:25.913841 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":26,"num_ops":1,"prepare.queue_time_us":113049,"prepare.run_cpu_time_us":773,"prepare.run_wall_time_us":444832,"replication_time_us":518158}]]}
W20260924 12:47:26.324082 7836 tablet_service.cc:2970] error setting up scanner fe6b4f9748ba49dc80731d14f384edf8 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.540s (maximum allowed 0.100s): new_scan_request { tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\357\271\324\325\006\"3\n\005slave\022*\n f7080e4ffff44a22a72b8608c75dca3e\020\001\030\001 \001(\001" signature: "<redacted>""\225\227\301p\373\026\334\232\225\005f\212x\357\265\2618\021`0\217\300\347\n\235Q\\t\253cU\353d\3312a\337\313\342\335X\221\266\t\221\034\225jV\301,:\201\203\335\022)\235}P\271\374\270\330" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b1e701df8a3f4c2b8c2f0ef68abb2436"
W20260924 12:47:26.567358 7836 tablet_service.cc:2970] error setting up scanner 124b0c44c4fa4320b47b10ddce31c4ae with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.783s (maximum allowed 0.100s): new_scan_request { tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\357\271\324\325\006\"3\n\005slave\022*\n f7080e4ffff44a22a72b8608c75dca3e\020\001\030\001 \001(\001" signature: "<redacted>""\225\227\301p\373\026\334\232\225\005f\212x\357\265\2618\021`0\217\300\347\n\235Q\\t\253cU\353d\3312a\337\313\342\335X\221\266\t\221\034\225jV\301,:\201\203\335\022)\235}P\271\374\270\330" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "dcfcbbc642ea47f3b43d5c78efc7a745"
I20260924 12:47:26.784684 8363 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:26.785066 7876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.5.77.194:39441 (request call id 3992) took 1001 ms. Trace:
I20260924 12:47:26.785218 7876 rpcz_store.cc:276] 0924 12:47:25.783575 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:25.783831 (+ 256us) service_pool.cc:227] Handling call
0924 12:47:25.784040 (+ 209us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 12:47:25.784153 (+ 113us) raft_consensus.cc:1520] Early marking committed up to index 39
0924 12:47:25.784474 (+ 321us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 12:47:25.784624 (+ 150us) write_op.cc:270] Start()
0924 12:47:26.783788 (+999164us) write_op.cc:276] Timestamp: P: 1790254044951061 usec, L: 0
0924 12:47:26.784487 (+ 699us) log.cc:842] Serialized 354 byte log entry
0924 12:47:26.784548 (+ 61us) raft_consensus.cc:1631] Marking committed up to 39
0924 12:47:26.784558 (+ 10us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 12:47:26.784592 (+ 34us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 12:47:26.784958 (+ 366us) raft_consensus.cc:1671] finished
0924 12:47:26.784960 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 12:47:26.785013 (+ 53us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:26.784342 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:26.784466 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:26.784473 (+ 7us) write_op.cc:435] Acquired schema lock
0924 12:47:26.784476 (+ 3us) tablet.cc:660] Decoding operations
0924 12:47:26.784573 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:26.784592 (+ 19us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:26.784596 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:26.784654 (+ 58us) tablet.cc:699] Row locks acquired
0924 12:47:26.784661 (+ 7us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":8,"prepare.queue_time_us":404,"prepare.run_cpu_time_us":460,"prepare.run_wall_time_us":459,"thread_start_us":286,"threads_started":1}]]}
I20260924 12:47:27.200028 8430 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:27.200273 8430 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:27.201059 8430 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:27.201546 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:27.201562 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:27.202148 8074 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:27.202430 8430 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:27.236860 8437 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=39, attempt_no=0}) took 1405 ms. Trace:
I20260924 12:47:27.236995 8437 rpcz_store.cc:276] 0924 12:47:25.831118 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:25.831185 (+ 67us) service_pool.cc:227] Handling call
0924 12:47:27.236813 (+1405628us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:25.831700 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:25.831799 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:25.831803 (+ 4us) write_op.cc:435] Acquired schema lock
0924 12:47:25.831805 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:25.831874 (+ 69us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:25.831883 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:25.831886 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:25.831927 (+ 41us) tablet.cc:699] Row locks acquired
0924 12:47:25.831929 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:25.831961 (+ 32us) write_op.cc:270] Start()
0924 12:47:26.545126 (+713165us) write_op.cc:276] Timestamp: P: 1790254045831922 usec, L: 0
0924 12:47:26.545132 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:47:26.545319 (+ 187us) log.cc:842] Serialized 354 byte log entry
0924 12:47:27.235677 (+690358us) op_driver.cc:444] REPLICATION: finished
0924 12:47:27.236269 (+ 592us) write_op.cc:301] APPLY: starting
0924 12:47:27.236288 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:27.236325 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:27.236359 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:27.236455 (+ 96us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:27.236484 (+ 29us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:27.236488 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:47:27.236597 (+ 109us) log.cc:842] Serialized 22 byte log entry
0924 12:47:27.236686 (+ 89us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:27.236702 (+ 16us) write_op.cc:454] Released schema lock
0924 12:47:27.236709 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":526,"num_ops":1,"prepare.queue_time_us":56,"prepare.run_cpu_time_us":632,"prepare.run_wall_time_us":713685,"replication_time_us":690504,"thread_start_us":329,"threads_started":1}]]}
I20260924 12:47:27.279134 8363 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:27.279348 8363 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:27.280085 8363 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:27.280578 8138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:27.280906 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:27.281452 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:27.711445 8430 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:27.711647 8430 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:27.712426 8430 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:27.712968 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:27.713089 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:27.713661 8073 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:27.714090 8430 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:27.731628 8437 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=40, attempt_no=0}) took 1815 ms. Trace:
I20260924 12:47:27.731752 8437 rpcz_store.cc:276] 0924 12:47:25.915805 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:25.915868 (+ 63us) service_pool.cc:227] Handling call
0924 12:47:27.731567 (+1815699us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:26.545429 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:26.545545 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:26.545551 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:47:26.545553 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:26.545657 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:26.545668 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:26.545673 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:26.545757 (+ 84us) tablet.cc:699] Row locks acquired
0924 12:47:26.545759 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:26.545801 (+ 42us) write_op.cc:270] Start()
0924 12:47:27.030974 (+485173us) write_op.cc:276] Timestamp: P: 1790254046545749 usec, L: 0
0924 12:47:27.030979 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:47:27.031230 (+ 251us) log.cc:842] Serialized 354 byte log entry
0924 12:47:27.731008 (+699778us) op_driver.cc:444] REPLICATION: finished
0924 12:47:27.731104 (+ 96us) write_op.cc:301] APPLY: starting
0924 12:47:27.731125 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:27.731171 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:27.731205 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:27.731286 (+ 81us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:27.731324 (+ 38us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:27.731328 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:47:27.731403 (+ 75us) log.cc:842] Serialized 22 byte log entry
0924 12:47:27.731469 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:27.731485 (+ 16us) write_op.cc:454] Released schema lock
0924 12:47:27.731491 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":29,"num_ops":1,"prepare.queue_time_us":629126,"prepare.run_cpu_time_us":804,"prepare.run_wall_time_us":485880,"replication_time_us":699995}]]}
I20260924 12:47:27.788537 8363 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:27.993752 8363 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:27.993963 8363 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:27.994727 8363 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:27.995211 8138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:27.995394 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:27.996279 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:28.356472 8430 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:28.356657 8430 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:28.357432 8430 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:28.358038 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:28.358168 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:28.358739 8074 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:28.885332 8363 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:28.907137 8450 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=42, attempt_no=0}) took 1173 ms. Trace:
I20260924 12:47:28.907262 8450 rpcz_store.cc:276] 0924 12:47:27.733442 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:27.733506 (+ 64us) service_pool.cc:227] Handling call
0924 12:47:28.907086 (+1173580us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:27.896998 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:27.897123 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:27.897130 (+ 7us) write_op.cc:435] Acquired schema lock
0924 12:47:27.897132 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:27.897220 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:27.897229 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:27.897233 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:27.897277 (+ 44us) tablet.cc:699] Row locks acquired
0924 12:47:27.897278 (+ 1us) write_op.cc:260] PREPARE: finished
0924 12:47:27.897318 (+ 40us) write_op.cc:270] Start()
0924 12:47:27.925475 (+ 28157us) write_op.cc:276] Timestamp: P: 1790254047897242 usec, L: 0
0924 12:47:27.925480 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:47:27.925784 (+ 304us) log.cc:842] Serialized 354 byte log entry
0924 12:47:28.905957 (+980173us) op_driver.cc:444] REPLICATION: finished
0924 12:47:28.906444 (+ 487us) write_op.cc:301] APPLY: starting
0924 12:47:28.906487 (+ 43us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:28.906533 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:28.906563 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:28.906655 (+ 92us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:28.906686 (+ 31us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:28.906690 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:47:28.906837 (+ 147us) log.cc:842] Serialized 22 byte log entry
0924 12:47:28.906951 (+ 114us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:28.906972 (+ 21us) write_op.cc:454] Released schema lock
0924 12:47:28.906980 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":421,"num_ops":1,"prepare.queue_time_us":163099,"prepare.run_cpu_time_us":804,"prepare.run_wall_time_us":28869,"replication_time_us":980437,"thread_start_us":276,"threads_started":1}]]}
W20260924 12:47:29.143173 7836 tablet_service.cc:2970] error setting up scanner b17fe883462848ce9486a78bcfa0bd37 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.255s (maximum allowed 0.100s): new_scan_request { tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\357\271\324\325\006\"3\n\005slave\022*\n f7080e4ffff44a22a72b8608c75dca3e\020\001\030\001 \001(\001" signature: "<redacted>""\225\227\301p\373\026\334\232\225\005f\212x\357\265\2618\021`0\217\300\347\n\235Q\\t\253cU\353d\3312a\337\313\342\335X\221\266\t\221\034\225jV\301,:\201\203\335\022)\235}P\271\374\270\330" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b1e701df8a3f4c2b8c2f0ef68abb2436"
I20260924 12:47:29.579072 8363 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:29.579272 8363 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:29.580008 8363 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:29.580579 8138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:29.580818 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:29.581425 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:29.581898 8363 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:29.583607 8457 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=43, attempt_no=0}) took 1551 ms. Trace:
I20260924 12:47:29.583751 8457 rpcz_store.cc:276] 0924 12:47:28.032162 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:28.032229 (+ 67us) service_pool.cc:227] Handling call
0924 12:47:29.583541 (+1551312us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:28.032743 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:28.032844 (+ 101us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:28.032848 (+ 4us) write_op.cc:435] Acquired schema lock
0924 12:47:28.032850 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:28.032924 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:28.032934 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:28.032938 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:28.032983 (+ 45us) tablet.cc:699] Row locks acquired
0924 12:47:28.032984 (+ 1us) write_op.cc:260] PREPARE: finished
0924 12:47:28.033018 (+ 34us) write_op.cc:270] Start()
0924 12:47:28.039199 (+ 6181us) write_op.cc:276] Timestamp: P: 1790254048032932 usec, L: 0
0924 12:47:28.039204 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:47:28.039416 (+ 212us) log.cc:842] Serialized 354 byte log entry
0924 12:47:29.582756 (+1543340us) op_driver.cc:444] REPLICATION: finished
0924 12:47:29.583111 (+ 355us) write_op.cc:301] APPLY: starting
0924 12:47:29.583128 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:29.583185 (+ 57us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:29.583211 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:29.583273 (+ 62us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:29.583298 (+ 25us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:29.583305 (+ 7us) write_op.cc:312] APPLY: finished
0924 12:47:29.583366 (+ 61us) log.cc:842] Serialized 22 byte log entry
0924 12:47:29.583435 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:29.583450 (+ 15us) write_op.cc:454] Released schema lock
0924 12:47:29.583457 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":298,"num_ops":1,"prepare.queue_time_us":52,"prepare.run_cpu_time_us":678,"prepare.run_wall_time_us":6775,"replication_time_us":1543528,"thread_start_us":202,"threads_started":1}]]}
I20260924 12:47:29.851550 8430 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:29.851979 8137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.5.77.194:53293 (request call id 6725) took 1493 ms. Trace:
I20260924 12:47:29.852121 8137 rpcz_store.cc:276] 0924 12:47:28.358093 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:28.358163 (+ 70us) service_pool.cc:227] Handling call
0924 12:47:28.358368 (+ 205us) raft_consensus.cc:1457] Updating replica for 2 ops
0924 12:47:28.358645 (+ 277us) raft_consensus.cc:1520] Early marking committed up to index 43
0924 12:47:28.359062 (+ 417us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0924 12:47:28.359207 (+ 145us) write_op.cc:270] Start()
0924 12:47:29.296381 (+937174us) write_op.cc:276] Timestamp: P: 1790254047897242 usec, L: 0
0924 12:47:29.296937 (+ 556us) write_op.cc:270] Start()
0924 12:47:29.850115 (+553178us) write_op.cc:276] Timestamp: P: 1790254048032932 usec, L: 0
0924 12:47:29.850959 (+ 844us) log.cc:842] Serialized 708 byte log entry
0924 12:47:29.851428 (+ 469us) raft_consensus.cc:1631] Marking committed up to 43
0924 12:47:29.851438 (+ 10us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 12:47:29.851469 (+ 31us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 12:47:29.851887 (+ 418us) raft_consensus.cc:1671] finished
0924 12:47:29.851889 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 12:47:29.851939 (+ 50us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:29.296892 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:29.297002 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:29.297007 (+ 5us) write_op.cc:435] Acquired schema lock
0924 12:47:29.297009 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:29.297093 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:29.297106 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:29.297109 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:29.297155 (+ 46us) tablet.cc:699] Row locks acquired
0924 12:47:29.297156 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0924 12:47:29.850773 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:29.850903 (+ 130us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:29.850908 (+ 5us) write_op.cc:435] Acquired schema lock
0924 12:47:29.850910 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:29.850993 (+ 83us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:29.851008 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:29.851013 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:29.851104 (+ 91us) tablet.cc:699] Row locks acquired
0924 12:47:29.851106 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":378,"threads_started":1,"wal-append.queue_time_us":466,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":369,"prepare.run_cpu_time_us":359,"prepare.run_wall_time_us":360,"thread_start_us":239,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":507,"prepare.run_cpu_time_us":456,"prepare.run_wall_time_us":457,"thread_start_us":394,"threads_started":1}]]}
I20260924 12:47:30.393761 8363 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:30.393968 8363 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:30.394942 8363 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:30.395401 8138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:30.395596 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:30.396136 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:30.396394 8363 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:30.444339 8466 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55890 (ReqId={client: 8344c72bdf7e4c3ebab2c70d6f997b6d, seq_no=44, attempt_no=0}) took 1535 ms. Trace:
I20260924 12:47:30.444494 8466 rpcz_store.cc:276] 0924 12:47:28.909007 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:28.909079 (+ 72us) service_pool.cc:227] Handling call
0924 12:47:30.444241 (+1535162us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:28.909902 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0121d1741e224a7ba39ced9c1fdb2b4b
0924 12:47:28.910083 (+ 181us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:28.910088 (+ 5us) write_op.cc:435] Acquired schema lock
0924 12:47:28.910089 (+ 1us) tablet.cc:660] Decoding operations
0924 12:47:28.910161 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:28.910170 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:28.910173 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:28.910241 (+ 68us) tablet.cc:699] Row locks acquired
0924 12:47:28.910243 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:28.910278 (+ 35us) write_op.cc:270] Start()
0924 12:47:29.869452 (+959174us) write_op.cc:276] Timestamp: P: 1790254048910232 usec, L: 0
0924 12:47:29.869457 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:47:29.869789 (+ 332us) log.cc:842] Serialized 354 byte log entry
0924 12:47:30.443085 (+573296us) op_driver.cc:444] REPLICATION: finished
0924 12:47:30.443629 (+ 544us) write_op.cc:301] APPLY: starting
0924 12:47:30.443650 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:30.443694 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:30.443722 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:30.443818 (+ 96us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:30.443859 (+ 41us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:30.443865 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:47:30.443962 (+ 97us) log.cc:842] Serialized 22 byte log entry
0924 12:47:30.444069 (+ 107us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:30.444091 (+ 22us) write_op.cc:454] Released schema lock
0924 12:47:30.444100 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":470,"num_ops":1,"prepare.queue_time_us":354,"prepare.run_cpu_time_us":883,"prepare.run_wall_time_us":959956,"replication_time_us":573596,"spinlock_wait_cycles":192768,"thread_start_us":534,"threads_started":2}]]}
I20260924 12:47:30.506491 8363 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:30.506734 8363 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:30.507445 8363 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:30.507957 8138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:30.508335 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:30.508998 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:30.509269 8363 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:30.842381 8430 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:30.842595 8430 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:30.843523 8430 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:30.843937 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:30.844347 7876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:30.844980 8074 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:31.170380 8430 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 12:47:31.186770 8232 scanner-internal.cc:458] Time spent opening tablet: real 31.426s user 0.009s sys 0.000s
I20260924 12:47:31.560513 8471 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:31.561044 8471 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:31.562129 8471 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:31.562758 8137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:31.563917 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:31.564814 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:31.565117 8471 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 12:47:31.965250 8231 scanner-internal.cc:458] Time spent opening tablet: real 32.204s user 0.009s sys 0.000s
I20260924 12:47:32.019268 8484 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:32.019570 8484 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:32.021656 8484 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:32.022307 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:32.022863 7876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:32.023453 8073 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:32.024065 8484 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:32.074656 8484 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:32.074873 8484 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:32.075594 8484 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:32.076295 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:32.079161 8499 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:32.079479 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:32.079967 8499 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:32.080344 7877 raft_consensus.cc:2454] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5592d2da7454940a8bade621094bf73 in term 1.
I20260924 12:47:32.080648 8499 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:32.080871 8073 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, a5592d2da7454940a8bade621094bf73; no voters: d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:32.081157 8137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:32.081411 8137 raft_consensus.cc:2454] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17f2a7fd399b4c8b9b96a1d8ec5d39e9 in term 1.
I20260924 12:47:32.081687 8484 raft_consensus.cc:2781] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 12:47:32.081611 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:32.081786 8484 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:32.081862 8484 raft_consensus.cc:3037] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:47:32.081856 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, a5592d2da7454940a8bade621094bf73; no voters:
I20260924 12:47:32.082352 8499 raft_consensus.cc:2781] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 12:47:32.082490 8499 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d0a6698cb7734148a8c2a168182e095e)
I20260924 12:47:32.082579 8499 raft_consensus.cc:3037] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:47:32.084692 8484 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:32.085180 8499 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:32.085906 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e"
I20260924 12:47:32.085996 8484 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 election: Requested vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:32.087003 8137 raft_consensus.cc:1226] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 2 FOLLOWER]: Rejecting Update request from peer d0a6698cb7734148a8c2a168182e095e for earlier term 1. Current term is 2. Ops: []
I20260924 12:47:32.087637 8463 consensus_queue.cc:1046] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 }, Status: INVALID_TERM, Last received: 1.47, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20260924 12:47:32.085798 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9"
I20260924 12:47:32.088518 7877 raft_consensus.cc:2379] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a5592d2da7454940a8bade621094bf73 in current term 2: Already voted for candidate 17f2a7fd399b4c8b9b96a1d8ec5d39e9 in this term.
I20260924 12:47:32.088949 8073 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5592d2da7454940a8bade621094bf73; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:32.089854 7877 raft_consensus.cc:1226] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 2 FOLLOWER]: Rejecting Update request from peer d0a6698cb7734148a8c2a168182e095e for earlier term 1. Current term is 2. Ops: []
I20260924 12:47:32.089890 8484 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 12:47:32.090363 8464 consensus_queue.cc:1046] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 }, Status: INVALID_TERM, Last received: 1.47, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20260924 12:47:32.090734 8137 raft_consensus.cc:1226] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 2 FOLLOWER]: Rejecting Update request from peer d0a6698cb7734148a8c2a168182e095e for earlier term 1. Current term is 2. Ops: []
I20260924 12:47:32.091214 8499 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 election: Requested vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:32.091328 8464 consensus_queue.cc:1046] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 }, Status: INVALID_TERM, Last received: 1.47, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20260924 12:47:32.091688 8137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73"
I20260924 12:47:32.091711 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e"
I20260924 12:47:32.091907 8137 raft_consensus.cc:2379] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 17f2a7fd399b4c8b9b96a1d8ec5d39e9 in current term 2: Already voted for candidate a5592d2da7454940a8bade621094bf73 in this term.
I20260924 12:47:32.092535 7877 raft_consensus.cc:1226] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 2 FOLLOWER]: Rejecting Update request from peer d0a6698cb7734148a8c2a168182e095e for earlier term 1. Current term is 2. Ops: []
I20260924 12:47:32.093253 8463 consensus_queue.cc:1046] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 }, Status: INVALID_TERM, Last received: 1.47, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20260924 12:47:32.093685 8137 raft_consensus.cc:1226] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 2 FOLLOWER]: Rejecting Update request from peer d0a6698cb7734148a8c2a168182e095e for earlier term 1. Current term is 2. Ops: []
I20260924 12:47:32.093674 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:32.094386 7877 raft_consensus.cc:1226] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 2 FOLLOWER]: Rejecting Update request from peer d0a6698cb7734148a8c2a168182e095e for earlier term 1. Current term is 2. Ops: []
I20260924 12:47:32.094482 8464 consensus_queue.cc:1046] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 }, Status: INVALID_TERM, Last received: 1.47, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20260924 12:47:32.094923 8464 consensus_queue.cc:1046] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 }, Status: INVALID_TERM, Last received: 1.47, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20260924 12:47:32.095443 8137 raft_consensus.cc:1226] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 2 FOLLOWER]: Rejecting Update request from peer d0a6698cb7734148a8c2a168182e095e for earlier term 1. Current term is 2. Ops: []
I20260924 12:47:32.095645 8499 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 12:47:32.095943 8464 consensus_queue.cc:1046] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 }, Status: INVALID_TERM, Last received: 1.47, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20260924 12:47:32.096925 7877 raft_consensus.cc:1226] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 2 FOLLOWER]: Rejecting Update request from peer d0a6698cb7734148a8c2a168182e095e for earlier term 1. Current term is 2. Ops: []
I20260924 12:47:32.097340 8464 consensus_queue.cc:1046] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 }, Status: INVALID_TERM, Last received: 1.47, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20260924 12:47:32.097534 8484 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:47:32.097700 8484 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:32.098260 8484 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:32.098613 8137 raft_consensus.cc:1226] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 2 FOLLOWER]: Rejecting Update request from peer d0a6698cb7734148a8c2a168182e095e for earlier term 1. Current term is 2. Ops: []
I20260924 12:47:32.098945 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 3 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:32.099189 7877 raft_consensus.cc:2454] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5592d2da7454940a8bade621094bf73 in term 2.
I20260924 12:47:32.099574 8463 consensus_queue.cc:1046] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 }, Status: INVALID_TERM, Last received: 1.47, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20260924 12:47:32.099742 8073 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, a5592d2da7454940a8bade621094bf73; no voters:
I20260924 12:47:32.099792 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 3 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:32.100030 8484 raft_consensus.cc:2781] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 12:47:32.100111 8484 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:47:32.100159 7877 raft_consensus.cc:1226] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 2 FOLLOWER]: Rejecting Update request from peer d0a6698cb7734148a8c2a168182e095e for earlier term 1. Current term is 2. Ops: []
I20260924 12:47:32.100214 8484 raft_consensus.cc:3037] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 2 FOLLOWER]: Advancing to term 3
I20260924 12:47:32.100739 8463 consensus_queue.cc:1046] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 }, Status: INVALID_TERM, Last received: 1.47, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20260924 12:47:32.103011 8484 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:32.103586 7877 raft_consensus.cc:1226] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 2 FOLLOWER]: Rejecting Update request from peer d0a6698cb7734148a8c2a168182e095e for earlier term 1. Current term is 2. Ops: []
I20260924 12:47:32.103962 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 3 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e"
I20260924 12:47:32.104130 8137 raft_consensus.cc:1226] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 3 FOLLOWER]: Rejecting Update request from peer d0a6698cb7734148a8c2a168182e095e for earlier term 1. Current term is 3. Ops: []
I20260924 12:47:32.104374 8464 consensus_queue.cc:1046] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 }, Status: INVALID_TERM, Last received: 1.47, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20260924 12:47:32.104538 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "a5592d2da7454940a8bade621094bf73" candidate_term: 3 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9"
I20260924 12:47:32.104656 8484 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 3 election: Requested vote from peers d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:32.104756 7877 raft_consensus.cc:3037] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 2 FOLLOWER]: Advancing to term 3
I20260924 12:47:32.105024 8463 consensus_queue.cc:1046] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 }, Status: INVALID_TERM, Last received: 1.47, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20260924 12:47:32.106769 8137 raft_consensus.cc:1226] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 3 FOLLOWER]: Rejecting Update request from peer d0a6698cb7734148a8c2a168182e095e for earlier term 1. Current term is 3. Ops: []
I20260924 12:47:32.107288 8464 consensus_queue.cc:1046] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 }, Status: INVALID_TERM, Last received: 1.47, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20260924 12:47:32.107793 7877 raft_consensus.cc:2454] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5592d2da7454940a8bade621094bf73 in term 3.
I20260924 12:47:32.108038 7876 raft_consensus.cc:1226] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 3 FOLLOWER]: Rejecting Update request from peer d0a6698cb7734148a8c2a168182e095e for earlier term 1. Current term is 3. Ops: []
I20260924 12:47:32.108192 8073 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, a5592d2da7454940a8bade621094bf73; no voters: d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:32.108414 8484 raft_consensus.cc:2781] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 12:47:32.108966 8484 raft_consensus.cc:686] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 3 LEADER]: Becoming Leader. State: Replica: a5592d2da7454940a8bade621094bf73, State: Running, Role: LEADER
I20260924 12:47:32.108909 8464 consensus_queue.cc:1046] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 }, Status: INVALID_TERM, Last received: 1.47, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20260924 12:47:32.109537 8484 consensus_queue.cc:237] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 1.47, Last appended by leader: 47, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:32.110551 7876 raft_consensus.cc:1226] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 3 FOLLOWER]: Rejecting Update request from peer d0a6698cb7734148a8c2a168182e095e for earlier term 1. Current term is 3. Ops: []
I20260924 12:47:32.111531 8118 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260924 12:47:32.112323 8137 raft_consensus.cc:1226] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [term 3 LEADER]: Rejecting Update request from peer d0a6698cb7734148a8c2a168182e095e for earlier term 1. Current term is 3. Ops: []
I20260924 12:47:32.113452 7736 catalog_manager.cc:5495] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 reported cstate change: term changed from 1 to 3, leader changed from d0a6698cb7734148a8c2a168182e095e (127.5.77.194) to a5592d2da7454940a8bade621094bf73 (127.5.77.195). New cstate: current_term: 3 leader_uuid: "a5592d2da7454940a8bade621094bf73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } health_report { overall_health: UNKNOWN } } }
I20260924 12:47:32.111943 8464 consensus_queue.cc:1046] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 }, Status: INVALID_TERM, Last received: 1.47, Next index: 48, Last known committed idx: 47, Time since last communication: 0.003s
I20260924 12:47:32.115199 7988 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260924 12:47:32.115711 8463 consensus_queue.cc:1046] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 }, Status: INVALID_TERM, Last received: 1.47, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20260924 12:47:32.115933 7876 raft_consensus.cc:1261] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 3 FOLLOWER]: Refusing update from remote peer a5592d2da7454940a8bade621094bf73: Log matching property violated. Preceding OpId in replica: term: 1 index: 47. Preceding OpId from leader: term: 3 index: 48. (index mismatch)
I20260924 12:47:32.115960 8463 consensus_queue.cc:1225] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [LEADER]: Peer a5592d2da7454940a8bade621094bf73 log is divergent from this leader: its last log entry 3.48 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 47
I20260924 12:47:32.116159 7857 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260924 12:47:32.116521 8008 raft_consensus.cc:3032] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [term 1 LEADER]: Stepping down as leader of term 1
I20260924 12:47:32.116647 8008 raft_consensus.cc:729] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d0a6698cb7734148a8c2a168182e095e, State: Running, Role: LEADER
I20260924 12:47:32.116652 8500 consensus_queue.cc:1035] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20260924 12:47:32.116990 8008 consensus_queue.cc:260] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 1.47, Last appended by leader: 47, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:32.117633 8008 raft_consensus.cc:3037] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [term 1 FOLLOWER]: Advancing to term 3
I20260924 12:47:32.121274 8008 raft_consensus.cc:1261] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [term 3 FOLLOWER]: Refusing update from remote peer a5592d2da7454940a8bade621094bf73: Log matching property violated. Preceding OpId in replica: term: 1 index: 47. Preceding OpId from leader: term: 3 index: 48. (index mismatch)
I20260924 12:47:32.122402 8463 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:47:32.122573 8463 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:32.122622 8484 consensus_queue.cc:1035] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20260924 12:47:32.123353 8463 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:32.124177 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "d0a6698cb7734148a8c2a168182e095e" candidate_term: 4 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:32.124569 7877 raft_consensus.cc:2396] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a6698cb7734148a8c2a168182e095e for term 4 because replica has last-logged OpId of term: 3 index: 48, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 47.
W20260924 12:47:32.124814 8470 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 3 is higher.
I20260924 12:47:32.124969 8137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "d0a6698cb7734148a8c2a168182e095e" candidate_term: 4 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:32.125756 8499 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a5592d2da7454940a8bade621094bf73)
I20260924 12:47:32.125936 8499 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:32.126595 8499 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:32.126981 7944 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a6698cb7734148a8c2a168182e095e; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, a5592d2da7454940a8bade621094bf73
I20260924 12:47:32.127349 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:32.132208 8137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:32.132835 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:32.133304 8499 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:47:32.140892 8499 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a5592d2da7454940a8bade621094bf73)
I20260924 12:47:32.141091 8499 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:32.141857 8499 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:32.142795 8137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:32.143422 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:32.144141 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:32.144668 8499 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:47:32.154366 8499 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a5592d2da7454940a8bade621094bf73)
I20260924 12:47:32.154624 8499 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:32.155282 8499 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:32.156067 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:32.156704 8137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:32.157476 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:32.157938 8499 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:47:32.170886 8499 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a5592d2da7454940a8bade621094bf73)
I20260924 12:47:32.171108 8499 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:32.171947 8499 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:32.172549 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:32.173172 8137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:32.174059 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:32.174454 8499 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:47:32.176914 8463 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:47:32.195690 8463 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a5592d2da7454940a8bade621094bf73)
I20260924 12:47:32.195896 8463 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:32.196751 8463 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:32.197145 8137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "d0a6698cb7734148a8c2a168182e095e" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:32.197635 7877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "d0a6698cb7734148a8c2a168182e095e" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" is_pre_election: true
I20260924 12:47:32.198380 7943 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a6698cb7734148a8c2a168182e095e; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, a5592d2da7454940a8bade621094bf73
I20260924 12:47:32.198683 8463 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
cb77ffc814ce46ab999a63f4f6c7fa14 | 127.5.77.254:33661 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.77.212:46531 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
17f2a7fd399b4c8b9b96a1d8ec5d39e9 | 127.5.77.193:39579 | HEALTHY | <none> | 0 | 0
a5592d2da7454940a8bade621094bf73 | 127.5.77.195:33859 | HEALTHY | <none> | 1 | 0
d0a6698cb7734148a8c2a168182e095e | 127.5.77.194:39283 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
consensus_inject_latency_ms_in_notifications | 50 | unsafe,hidden | all 3 server(s) checked
inject_latency_ms_before_starting_op | 1000 | hidden,advanced | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.5.77.193 | experimental | 127.5.77.193:39579
local_ip_for_outbound_sockets | 127.5.77.194 | experimental | 127.5.77.194:39283
local_ip_for_outbound_sockets | 127.5.77.195 | experimental | 127.5.77.195:33859
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.5.77.193:39579
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.77.194:39283
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.77.195:33859
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.77.212:46531 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 0121d1741e224a7ba39ced9c1fdb2b4b of table 'test-workload' is conflicted: 3 replicas' active configs disagree with the leader master's
a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859): RUNNING
d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283): RUNNING [LEADER]
17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579): RUNNING
All reported replicas are:
A = a5592d2da7454940a8bade621094bf73
B = d0a6698cb7734148a8c2a168182e095e
C = 17f2a7fd399b4c8b9b96a1d8ec5d39e9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A* B C | 3 | -1 | Yes
B | A B C | 3 | -1 | Yes
C | A* B C | 3 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260924 12:47:32.216033 5431 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260924 12:47:32.480517 7857 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260924 12:47:32.506549 8118 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260924 12:47:32.514401 7988 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
cb77ffc814ce46ab999a63f4f6c7fa14 | 127.5.77.254:33661 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.77.212:46531 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
17f2a7fd399b4c8b9b96a1d8ec5d39e9 | 127.5.77.193:39579 | HEALTHY | <none> | 0 | 0
a5592d2da7454940a8bade621094bf73 | 127.5.77.195:33859 | HEALTHY | <none> | 1 | 0
d0a6698cb7734148a8c2a168182e095e | 127.5.77.194:39283 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
consensus_inject_latency_ms_in_notifications | 50 | unsafe,hidden | all 3 server(s) checked
inject_latency_ms_before_starting_op | 1000 | hidden,advanced | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.5.77.193 | experimental | 127.5.77.193:39579
local_ip_for_outbound_sockets | 127.5.77.194 | experimental | 127.5.77.194:39283
local_ip_for_outbound_sockets | 127.5.77.195 | experimental | 127.5.77.195:33859
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.5.77.193:39579
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.77.194:39283
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.77.195:33859
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.77.212:46531 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260924 12:47:32.558149 5431 log_verifier.cc:126] Checking tablet 0121d1741e224a7ba39ced9c1fdb2b4b
I20260924 12:47:32.559407 8499 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a5592d2da7454940a8bade621094bf73)
I20260924 12:47:32.559617 8499 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:32.560233 8499 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283)
I20260924 12:47:32.560873 8137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
I20260924 12:47:32.561860 8007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "d0a6698cb7734148a8c2a168182e095e" is_pre_election: true
I20260924 12:47:32.562583 7813 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9; no voters: a5592d2da7454940a8bade621094bf73, d0a6698cb7734148a8c2a168182e095e
I20260924 12:47:32.562855 8499 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P 17f2a7fd399b4c8b9b96a1d8ec5d39e9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 12:47:32.623761 5431 log_verifier.cc:177] Verified matching terms for 48 ops in tablet 0121d1741e224a7ba39ced9c1fdb2b4b
I20260924 12:47:32.656340 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 7795
I20260924 12:47:32.664343 7917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:47:32.847728 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 7795
I20260924 12:47:32.853248 8470 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a5592d2da7454940a8bade621094bf73)
I20260924 12:47:32.853474 8470 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:32.854211 8470 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:32.854663 8137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "d0a6698cb7734148a8c2a168182e095e" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
W20260924 12:47:32.860531 7943 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.5.77.193:39579 timed out after 0.006s (SENT) [suppressed 1 similar messages]
W20260924 12:47:32.867946 7943 leader_election.cc:334] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579): Timed out: RequestConsensusVote RPC to 127.5.77.193:39579 timed out after 0.006s (SENT)
I20260924 12:47:32.868112 7943 leader_election.cc:302] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a6698cb7734148a8c2a168182e095e; no voters: 17f2a7fd399b4c8b9b96a1d8ec5d39e9, a5592d2da7454940a8bade621094bf73
I20260924 12:47:32.868564 8470 raft_consensus.cc:2726] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 12:47:32.877985 8073 connection.cc:535] client connection to 127.5.77.193:39579 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 12:47:32.878188 8073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 1 similar messages]
W20260924 12:47:32.878903 7943 connection.cc:535] client connection to 127.5.77.193:39579 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 12:47:32.879123 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 7926
W20260924 12:47:32.881062 8073 consensus_peers.cc:487] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 -> Peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579): Couldn't send request to peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:39579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 12:47:32.886984 8048 generic_service.cc:196] Checking for leaks (request via RPC)
W20260924 12:47:32.894394 8073 consensus_peers.cc:487] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 -> Peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579): Couldn't send request to peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:39579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260924 12:47:32.906137 8073 consensus_peers.cc:487] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 -> Peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579): Couldn't send request to peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:39579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260924 12:47:32.918745 8073 consensus_peers.cc:487] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 -> Peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579): Couldn't send request to peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:39579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260924 12:47:32.928879 8073 consensus_peers.cc:487] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 -> Peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579): Couldn't send request to peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:39579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260924 12:47:32.936856 8073 consensus_peers.cc:487] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 -> Peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579): Couldn't send request to peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:39579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260924 12:47:32.948438 8073 consensus_peers.cc:487] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 -> Peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579): Couldn't send request to peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:39579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260924 12:47:32.959790 8073 consensus_peers.cc:487] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 -> Peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579): Couldn't send request to peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:39579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260924 12:47:32.971406 8073 consensus_peers.cc:487] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 -> Peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579): Couldn't send request to peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:39579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260924 12:47:32.982843 8073 consensus_peers.cc:487] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 -> Peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579): Couldn't send request to peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:39579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260924 12:47:32.994136 8073 consensus_peers.cc:487] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 -> Peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579): Couldn't send request to peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:39579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260924 12:47:33.004541 8073 consensus_peers.cc:487] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 -> Peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579): Couldn't send request to peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:39579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260924 12:47:33.016384 8073 consensus_peers.cc:487] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 -> Peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579): Couldn't send request to peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:39579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260924 12:47:33.027709 8073 consensus_peers.cc:487] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 -> Peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579): Couldn't send request to peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:39579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260924 12:47:33.038942 8073 consensus_peers.cc:487] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 -> Peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579): Couldn't send request to peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:39579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20260924 12:47:33.050333 8073 consensus_peers.cc:487] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 -> Peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579): Couldn't send request to peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:39579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20260924 12:47:33.061843 8073 consensus_peers.cc:487] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 -> Peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579): Couldn't send request to peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:39579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20260924 12:47:33.072232 8470 raft_consensus.cc:482] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a5592d2da7454940a8bade621094bf73)
I20260924 12:47:33.072481 8470 raft_consensus.cc:504] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5592d2da7454940a8bade621094bf73" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 33859 } } peers { permanent_uuid: "d0a6698cb7734148a8c2a168182e095e" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39283 } } peers { permanent_uuid: "17f2a7fd399b4c8b9b96a1d8ec5d39e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 39579 } }
I20260924 12:47:33.073112 8470 leader_election.cc:288] T 0121d1741e224a7ba39ced9c1fdb2b4b P d0a6698cb7734148a8c2a168182e095e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a5592d2da7454940a8bade621094bf73 (127.5.77.195:33859), 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579)
I20260924 12:47:33.073181 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 7926
I20260924 12:47:33.074237 8137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0121d1741e224a7ba39ced9c1fdb2b4b" candidate_uuid: "d0a6698cb7734148a8c2a168182e095e" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "a5592d2da7454940a8bade621094bf73" is_pre_election: true
W20260924 12:47:33.079704 8073 consensus_peers.cc:487] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 -> Peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579): Couldn't send request to peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:39579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20260924 12:47:33.091462 8073 consensus_peers.cc:487] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 -> Peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579): Couldn't send request to peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:39579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20260924 12:47:33.096238 8071 connection.cc:535] server connection from 127.5.77.194:53293 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 12:47:33.096503 8074 connection.cc:535] client connection to 127.5.77.194:39283 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 12:47:33.096942 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 8056
W20260924 12:47:33.097329 8074 consensus_peers.cc:487] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 -> Peer d0a6698cb7734148a8c2a168182e095e (127.5.77.194:39283): Couldn't send request to peer d0a6698cb7734148a8c2a168182e095e. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.194:39283: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 12:47:33.103525 8073 consensus_peers.cc:487] T 0121d1741e224a7ba39ced9c1fdb2b4b P a5592d2da7454940a8bade621094bf73 -> Peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9 (127.5.77.193:39579): Couldn't send request to peer 17f2a7fd399b4c8b9b96a1d8ec5d39e9. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:39579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20260924 12:47:33.104970 8178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:47:33.243777 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 8056
I20260924 12:47:33.265781 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 7705
I20260924 12:47:33.266839 7766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:47:33.375180 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 7705
2026-09-24T12:47:33Z chronyd exiting
[ OK ] RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency (35823 ms)
[ RUN ] RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys
I20260924 12:47:33.412652 5431 test_util.cc:268] Using random seed: 15290113
I20260924 12:47:33.414148 5431 ts_itest-base.cc:112] Starting cluster with:
I20260924 12:47:33.414224 5431 ts_itest-base.cc:113] --------------
I20260924 12:47:33.414273 5431 ts_itest-base.cc:114] 3 tablet servers
I20260924 12:47:33.414332 5431 ts_itest-base.cc:115] 3 replicas per TS
I20260924 12:47:33.414377 5431 ts_itest-base.cc:116] --------------
2026-09-24T12:47:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:47:33Z Disabled control of system clock
I20260924 12:47:33.438482 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.77.254:32945
--webserver_interface=127.5.77.254
--webserver_port=0
--builtin_ntp_servers=127.5.77.212:41043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.77.254:32945 with env {}
W20260924 12:47:33.738530 8559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:47:33.738855 8559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:47:33.738976 8559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:47:33.746210 8559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:47:33.746325 8559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:47:33.746390 8559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:47:33.746475 8559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:47:33.755515 8559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:41043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.77.254:32945
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.77.254:32945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.77.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:47:33.756788 8559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:47:33.758510 8559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:47:33.769814 8567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:47:33.770076 8559 server_base.cc:1034] running on GCE node
W20260924 12:47:33.769863 8565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:33.769832 8564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:47:33.771162 8559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:47:33.772543 8559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:47:33.773748 8559 hybrid_clock.cc:648] HybridClock initialized: now 1790254053773678 us; error 53 us; skew 500 ppm
I20260924 12:47:33.774044 8559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:47:33.776113 8559 webserver.cc:460] Webserver started at http://127.5.77.254:35551/ using document root <none> and password file <none>
I20260924 12:47:33.776772 8559 fs_manager.cc:359] Metadata directory not provided
I20260924 12:47:33.776883 8559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:47:33.777153 8559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:47:33.779716 8559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "e0984e14de3041ddb9233bd3b9f3c1f2"
format_stamp: "Formatted at 2026-09-24 12:47:33 on dist-test-slave-b8q3"
I20260924 12:47:33.780418 8559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "e0984e14de3041ddb9233bd3b9f3c1f2"
format_stamp: "Formatted at 2026-09-24 12:47:33 on dist-test-slave-b8q3"
I20260924 12:47:33.785187 8559 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260924 12:47:33.788291 8573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:33.789520 8559 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 12:47:33.789660 8559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e0984e14de3041ddb9233bd3b9f3c1f2"
format_stamp: "Formatted at 2026-09-24 12:47:33 on dist-test-slave-b8q3"
I20260924 12:47:33.789830 8559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:47:33.820204 8559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:47:33.821028 8559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:47:33.821309 8559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:47:33.838079 8559 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.254:32945
I20260924 12:47:33.838101 8624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.254:32945 every 8 connection(s)
I20260924 12:47:33.839411 8559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260924 12:47:33.842291 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 8559
I20260924 12:47:33.842492 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260924 12:47:33.843149 8625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:47:33.854722 8625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0984e14de3041ddb9233bd3b9f3c1f2: Bootstrap starting.
I20260924 12:47:33.858283 8625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0984e14de3041ddb9233bd3b9f3c1f2: Neither blocks nor log segments found. Creating new log.
I20260924 12:47:33.859290 8625 log.cc:824] T 00000000000000000000000000000000 P e0984e14de3041ddb9233bd3b9f3c1f2: Log is configured to *not* fsync() on all Append() calls
I20260924 12:47:33.861707 8625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0984e14de3041ddb9233bd3b9f3c1f2: No bootstrap required, opened a new log
I20260924 12:47:33.868886 8625 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0984e14de3041ddb9233bd3b9f3c1f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0984e14de3041ddb9233bd3b9f3c1f2" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 32945 } }
I20260924 12:47:33.869251 8625 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0984e14de3041ddb9233bd3b9f3c1f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:47:33.869352 8625 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0984e14de3041ddb9233bd3b9f3c1f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0984e14de3041ddb9233bd3b9f3c1f2, State: Initialized, Role: FOLLOWER
I20260924 12:47:33.869923 8625 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0984e14de3041ddb9233bd3b9f3c1f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0984e14de3041ddb9233bd3b9f3c1f2" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 32945 } }
I20260924 12:47:33.870160 8625 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0984e14de3041ddb9233bd3b9f3c1f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:47:33.870298 8625 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0984e14de3041ddb9233bd3b9f3c1f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:47:33.870483 8625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0984e14de3041ddb9233bd3b9f3c1f2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:47:33.872538 8625 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0984e14de3041ddb9233bd3b9f3c1f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0984e14de3041ddb9233bd3b9f3c1f2" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 32945 } }
I20260924 12:47:33.873004 8625 leader_election.cc:302] T 00000000000000000000000000000000 P e0984e14de3041ddb9233bd3b9f3c1f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0984e14de3041ddb9233bd3b9f3c1f2; no voters:
I20260924 12:47:33.873605 8625 leader_election.cc:288] T 00000000000000000000000000000000 P e0984e14de3041ddb9233bd3b9f3c1f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:47:33.873797 8630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0984e14de3041ddb9233bd3b9f3c1f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:47:33.874395 8630 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0984e14de3041ddb9233bd3b9f3c1f2 [term 1 LEADER]: Becoming Leader. State: Replica: e0984e14de3041ddb9233bd3b9f3c1f2, State: Running, Role: LEADER
I20260924 12:47:33.874922 8630 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0984e14de3041ddb9233bd3b9f3c1f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0984e14de3041ddb9233bd3b9f3c1f2" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 32945 } }
I20260924 12:47:33.875509 8625 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0984e14de3041ddb9233bd3b9f3c1f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:47:33.877756 8632 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0984e14de3041ddb9233bd3b9f3c1f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0984e14de3041ddb9233bd3b9f3c1f2. Latest consensus state: current_term: 1 leader_uuid: "e0984e14de3041ddb9233bd3b9f3c1f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0984e14de3041ddb9233bd3b9f3c1f2" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 32945 } } }
I20260924 12:47:33.878126 8632 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0984e14de3041ddb9233bd3b9f3c1f2 [sys.catalog]: This master's current role is: LEADER
I20260924 12:47:33.878396 8631 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0984e14de3041ddb9233bd3b9f3c1f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0984e14de3041ddb9233bd3b9f3c1f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0984e14de3041ddb9233bd3b9f3c1f2" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 32945 } } }
I20260924 12:47:33.878736 8631 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0984e14de3041ddb9233bd3b9f3c1f2 [sys.catalog]: This master's current role is: LEADER
I20260924 12:47:33.880054 8638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:47:33.885236 8638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:47:33.893711 8638 catalog_manager.cc:1347] Generated new cluster ID: 7b495c38602a4322b442e619d9b8d62a
I20260924 12:47:33.893810 8638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:47:33.921825 8638 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:47:33.923674 8638 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:47:33.932152 8638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0984e14de3041ddb9233bd3b9f3c1f2: Generated new TSK 0
I20260924 12:47:33.933099 8638 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:47:33.949998 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.193:0
--local_ip_for_outbound_sockets=127.5.77.193
--webserver_interface=127.5.77.193
--webserver_port=0
--tserver_master_addrs=127.5.77.254:32945
--builtin_ntp_servers=127.5.77.212:41043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
W20260924 12:47:34.269536 8649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:47:34.269856 8649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:47:34.269984 8649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:47:34.277505 8649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:47:34.277720 8649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.193
I20260924 12:47:34.287169 8649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:41043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.77.193
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.5.77.254:32945
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.193
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:47:34.288481 8649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:47:34.290215 8649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:47:34.302939 8654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:34.303032 8655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:34.303068 8657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:47:34.304004 8649 server_base.cc:1034] running on GCE node
I20260924 12:47:34.304703 8649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:47:34.305688 8649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:47:34.306917 8649 hybrid_clock.cc:648] HybridClock initialized: now 1790254054306835 us; error 49 us; skew 500 ppm
I20260924 12:47:34.307216 8649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:47:34.309437 8649 webserver.cc:460] Webserver started at http://127.5.77.193:37047/ using document root <none> and password file <none>
I20260924 12:47:34.310111 8649 fs_manager.cc:359] Metadata directory not provided
I20260924 12:47:34.310227 8649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:47:34.310552 8649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:47:34.313285 8649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "0764fc96f6b048d6af98e1e628eef36b"
format_stamp: "Formatted at 2026-09-24 12:47:34 on dist-test-slave-b8q3"
I20260924 12:47:34.314102 8649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "0764fc96f6b048d6af98e1e628eef36b"
format_stamp: "Formatted at 2026-09-24 12:47:34 on dist-test-slave-b8q3"
I20260924 12:47:34.319386 8649 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260924 12:47:34.322832 8663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:34.324172 8649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 12:47:34.324347 8649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "0764fc96f6b048d6af98e1e628eef36b"
format_stamp: "Formatted at 2026-09-24 12:47:34 on dist-test-slave-b8q3"
I20260924 12:47:34.324548 8649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:47:34.359325 8649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:47:34.360201 8649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:47:34.360486 8649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:47:34.361438 8649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:47:34.363070 8649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:47:34.363152 8649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:34.363270 8649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:47:34.363322 8649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:34.393754 8649 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.193:41413
I20260924 12:47:34.393810 8775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.193:41413 every 8 connection(s)
I20260924 12:47:34.395059 8649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260924 12:47:34.400357 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 8649
I20260924 12:47:34.400543 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260924 12:47:34.403908 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.194:0
--local_ip_for_outbound_sockets=127.5.77.194
--webserver_interface=127.5.77.194
--webserver_port=0
--tserver_master_addrs=127.5.77.254:32945
--builtin_ntp_servers=127.5.77.212:41043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
I20260924 12:47:34.410640 8776 heartbeater.cc:347] Connected to a master server at 127.5.77.254:32945
I20260924 12:47:34.411015 8776 heartbeater.cc:464] Registering TS with master...
I20260924 12:47:34.411769 8776 heartbeater.cc:511] Master 127.5.77.254:32945 requested a full tablet report, sending...
I20260924 12:47:34.414028 8590 ts_manager.cc:194] Registered new tserver with Master: 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:47:34.415971 8590 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.193:38815
W20260924 12:47:34.722839 8780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:47:34.723143 8780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:47:34.723227 8780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:47:34.730695 8780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:47:34.730880 8780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.194
I20260924 12:47:34.740453 8780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:41043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.77.194
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.5.77.254:32945
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.194
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:47:34.741710 8780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:47:34.743409 8780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:47:34.755898 8785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:34.755863 8786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:34.755833 8788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:47:34.756827 8780 server_base.cc:1034] running on GCE node
I20260924 12:47:34.757390 8780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:47:34.758323 8780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:47:34.759539 8780 hybrid_clock.cc:648] HybridClock initialized: now 1790254054759456 us; error 79 us; skew 500 ppm
I20260924 12:47:34.759855 8780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:47:34.762022 8780 webserver.cc:460] Webserver started at http://127.5.77.194:43651/ using document root <none> and password file <none>
I20260924 12:47:34.762735 8780 fs_manager.cc:359] Metadata directory not provided
I20260924 12:47:34.762847 8780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:47:34.763116 8780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:47:34.765738 8780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "34e93f57f57f41d49de37a374ea07bb9"
format_stamp: "Formatted at 2026-09-24 12:47:34 on dist-test-slave-b8q3"
I20260924 12:47:34.766451 8780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "34e93f57f57f41d49de37a374ea07bb9"
format_stamp: "Formatted at 2026-09-24 12:47:34 on dist-test-slave-b8q3"
I20260924 12:47:34.771535 8780 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 12:47:34.774927 8794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:34.776296 8780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 12:47:34.776438 8780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "34e93f57f57f41d49de37a374ea07bb9"
format_stamp: "Formatted at 2026-09-24 12:47:34 on dist-test-slave-b8q3"
I20260924 12:47:34.776610 8780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:47:34.799469 8780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:47:34.800298 8780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:47:34.800565 8780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:47:34.801479 8780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:47:34.802946 8780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:47:34.803021 8780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:34.803143 8780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:47:34.803193 8780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:34.832597 8780 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.194:45615
I20260924 12:47:34.832603 8906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.194:45615 every 8 connection(s)
I20260924 12:47:34.833856 8780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260924 12:47:34.835563 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 8780
I20260924 12:47:34.835726 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260924 12:47:34.839015 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.195:0
--local_ip_for_outbound_sockets=127.5.77.195
--webserver_interface=127.5.77.195
--webserver_port=0
--tserver_master_addrs=127.5.77.254:32945
--builtin_ntp_servers=127.5.77.212:41043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
I20260924 12:47:34.849114 8907 heartbeater.cc:347] Connected to a master server at 127.5.77.254:32945
I20260924 12:47:34.849418 8907 heartbeater.cc:464] Registering TS with master...
I20260924 12:47:34.850163 8907 heartbeater.cc:511] Master 127.5.77.254:32945 requested a full tablet report, sending...
I20260924 12:47:34.851585 8590 ts_manager.cc:194] Registered new tserver with Master: 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:47:34.852389 8590 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.194:59047
W20260924 12:47:35.175009 8910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:47:35.175328 8910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:47:35.175426 8910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:47:35.182878 8910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:47:35.183060 8910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.195
I20260924 12:47:35.192133 8910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:41043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.77.195
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.5.77.254:32945
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.195
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:47:35.193444 8910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:47:35.195283 8910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:47:35.207790 8916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:35.207859 8918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:35.207813 8915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:47:35.208711 8910 server_base.cc:1034] running on GCE node
I20260924 12:47:35.209297 8910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:47:35.210208 8910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:47:35.211426 8910 hybrid_clock.cc:648] HybridClock initialized: now 1790254055211333 us; error 74 us; skew 500 ppm
I20260924 12:47:35.211733 8910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:47:35.213954 8910 webserver.cc:460] Webserver started at http://127.5.77.195:44213/ using document root <none> and password file <none>
I20260924 12:47:35.214689 8910 fs_manager.cc:359] Metadata directory not provided
I20260924 12:47:35.214794 8910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:47:35.215067 8910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:47:35.217624 8910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "4b04ce5d5bd546d880ced66f9b15decd"
format_stamp: "Formatted at 2026-09-24 12:47:35 on dist-test-slave-b8q3"
I20260924 12:47:35.218334 8910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "4b04ce5d5bd546d880ced66f9b15decd"
format_stamp: "Formatted at 2026-09-24 12:47:35 on dist-test-slave-b8q3"
I20260924 12:47:35.223701 8910 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 12:47:35.227185 8924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:35.228645 8910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:47:35.228796 8910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "4b04ce5d5bd546d880ced66f9b15decd"
format_stamp: "Formatted at 2026-09-24 12:47:35 on dist-test-slave-b8q3"
I20260924 12:47:35.229003 8910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:47:35.253484 8910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:47:35.254329 8910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:47:35.254647 8910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:47:35.255538 8910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:47:35.256965 8910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:47:35.257037 8910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:35.257150 8910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:47:35.257201 8910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:35.286294 8910 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.195:38151
I20260924 12:47:35.286338 9036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.195:38151 every 8 connection(s)
I20260924 12:47:35.287600 8910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260924 12:47:35.290697 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 8910
I20260924 12:47:35.290877 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260924 12:47:35.300979 9037 heartbeater.cc:347] Connected to a master server at 127.5.77.254:32945
I20260924 12:47:35.301252 9037 heartbeater.cc:464] Registering TS with master...
I20260924 12:47:35.301923 9037 heartbeater.cc:511] Master 127.5.77.254:32945 requested a full tablet report, sending...
I20260924 12:47:35.303221 8590 ts_manager.cc:194] Registered new tserver with Master: 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151)
I20260924 12:47:35.303956 8590 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.195:50841
I20260924 12:47:35.306348 5431 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:47:35.309387 5431 test_util.cc:268] Using random seed: 17186868
I20260924 12:47:35.328526 8590 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44128:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:47:35.330049 8590 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:47:35.367921 8842 tablet_service.cc:1467] Processing CreateTablet for tablet b077e0d6af50421c8292518a044c561e (DEFAULT_TABLE table=test-workload [id=94c93e68658b4a4f959e888eca18878d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:47:35.367921 8972 tablet_service.cc:1467] Processing CreateTablet for tablet b077e0d6af50421c8292518a044c561e (DEFAULT_TABLE table=test-workload [id=94c93e68658b4a4f959e888eca18878d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:47:35.368242 8711 tablet_service.cc:1467] Processing CreateTablet for tablet b077e0d6af50421c8292518a044c561e (DEFAULT_TABLE table=test-workload [id=94c93e68658b4a4f959e888eca18878d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:47:35.369369 8711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b077e0d6af50421c8292518a044c561e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:47:35.369379 8972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b077e0d6af50421c8292518a044c561e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:47:35.369379 8842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b077e0d6af50421c8292518a044c561e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:47:35.380916 9055 tablet_bootstrap.cc:492] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b: Bootstrap starting.
I20260924 12:47:35.382939 9056 tablet_bootstrap.cc:492] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd: Bootstrap starting.
I20260924 12:47:35.384239 9057 tablet_bootstrap.cc:492] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9: Bootstrap starting.
I20260924 12:47:35.386590 9055 tablet_bootstrap.cc:654] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b: Neither blocks nor log segments found. Creating new log.
I20260924 12:47:35.386590 9056 tablet_bootstrap.cc:654] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd: Neither blocks nor log segments found. Creating new log.
I20260924 12:47:35.386993 9057 tablet_bootstrap.cc:654] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9: Neither blocks nor log segments found. Creating new log.
I20260924 12:47:35.387888 9057 log.cc:824] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9: Log is configured to *not* fsync() on all Append() calls
I20260924 12:47:35.387883 9055 log.cc:824] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b: Log is configured to *not* fsync() on all Append() calls
I20260924 12:47:35.387883 9056 log.cc:824] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd: Log is configured to *not* fsync() on all Append() calls
I20260924 12:47:35.390533 9057 tablet_bootstrap.cc:492] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9: No bootstrap required, opened a new log
I20260924 12:47:35.390793 9057 ts_tablet_manager.cc:1397] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260924 12:47:35.395848 9056 tablet_bootstrap.cc:492] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd: No bootstrap required, opened a new log
I20260924 12:47:35.395846 9055 tablet_bootstrap.cc:492] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b: No bootstrap required, opened a new log
I20260924 12:47:35.396157 9055 ts_tablet_manager.cc:1397] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.006s
I20260924 12:47:35.396272 9056 ts_tablet_manager.cc:1397] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20260924 12:47:35.400035 9057 raft_consensus.cc:348] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:35.400470 9057 raft_consensus.cc:374] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:47:35.400565 9057 raft_consensus.cc:729] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34e93f57f57f41d49de37a374ea07bb9, State: Initialized, Role: FOLLOWER
I20260924 12:47:35.401379 9057 consensus_queue.cc:260] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:35.407698 9056 raft_consensus.cc:348] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:35.408239 9056 raft_consensus.cc:374] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:47:35.408339 9056 raft_consensus.cc:729] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b04ce5d5bd546d880ced66f9b15decd, State: Initialized, Role: FOLLOWER
I20260924 12:47:35.409312 9056 consensus_queue.cc:260] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:35.408886 9055 raft_consensus.cc:348] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:35.409744 9055 raft_consensus.cc:374] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:47:35.409849 9055 raft_consensus.cc:729] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0764fc96f6b048d6af98e1e628eef36b, State: Initialized, Role: FOLLOWER
I20260924 12:47:35.410562 9055 consensus_queue.cc:260] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:35.411046 9062 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:47:35.411402 9062 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:35.418776 9063 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:47:35.419220 8776 heartbeater.cc:503] Master 127.5.77.254:32945 was elected leader, sending a full tablet report...
I20260924 12:47:35.419073 9063 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
W20260924 12:47:35.438593 8928 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.5.77.193:41413 for RPC RequestConsensusVote timed out after 0.008s (ON_OUTBOUND_QUEUE)
I20260924 12:47:35.443969 9062 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:47:35.444825 9037 heartbeater.cc:503] Master 127.5.77.254:32945 was elected leader, sending a full tablet report...
I20260924 12:47:35.449905 9066 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:47:35.450349 9066 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:35.460172 8862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "4b04ce5d5bd546d880ced66f9b15decd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9" is_pre_election: true
I20260924 12:47:35.460760 8862 raft_consensus.cc:2454] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b04ce5d5bd546d880ced66f9b15decd in term 0.
I20260924 12:47:35.461591 8927 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9, 4b04ce5d5bd546d880ced66f9b15decd; no voters:
I20260924 12:47:35.463372 8731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "4b04ce5d5bd546d880ced66f9b15decd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:47:35.463847 8731 raft_consensus.cc:2454] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b04ce5d5bd546d880ced66f9b15decd in term 0.
I20260924 12:47:35.464536 9057 ts_tablet_manager.cc:1428] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9: Time spent starting tablet: real 0.074s user 0.011s sys 0.004s
W20260924 12:47:35.464918 8795 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.5.77.195:38151 for RPC RequestConsensusVote timed out after 0.007s (ON_OUTBOUND_QUEUE)
I20260924 12:47:35.465135 9062 raft_consensus.cc:2781] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:47:35.465379 9062 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:47:35.465540 9062 raft_consensus.cc:3037] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:47:35.467042 8907 heartbeater.cc:503] Master 127.5.77.254:32945 was elected leader, sending a full tablet report...
I20260924 12:47:35.472566 9062 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:35.477366 8731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "4b04ce5d5bd546d880ced66f9b15decd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b"
I20260924 12:47:35.477649 8731 raft_consensus.cc:3037] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:47:35.478994 9063 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:47:35.481000 8731 raft_consensus.cc:2454] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b04ce5d5bd546d880ced66f9b15decd in term 1.
I20260924 12:47:35.481505 8928 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd; no voters:
I20260924 12:47:35.484629 9064 raft_consensus.cc:2781] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:47:35.485759 9056 ts_tablet_manager.cc:1428] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd: Time spent starting tablet: real 0.089s user 0.046s sys 0.033s
I20260924 12:47:35.486580 9064 raft_consensus.cc:686] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd [term 1 LEADER]: Becoming Leader. State: Replica: 4b04ce5d5bd546d880ced66f9b15decd, State: Running, Role: LEADER
I20260924 12:47:35.487200 9064 consensus_queue.cc:237] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:35.491289 9066 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
W20260924 12:47:35.491987 8928 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Couldn't send request to peer 0764fc96f6b048d6af98e1e628eef36b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260924 12:47:35.495038 9062 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd [CANDIDATE]: Term 1 election: Requested vote from peers 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:47:35.496887 9055 ts_tablet_manager.cc:1428] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b: Time spent starting tablet: real 0.101s user 0.059s sys 0.038s
W20260924 12:47:35.499701 8666 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.5.77.194:45615 for RPC RequestConsensusVote timed out after 0.007s (ON_OUTBOUND_QUEUE)
I20260924 12:47:35.500114 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:35.506932 8862 raft_consensus.cc:3037] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 0 FOLLOWER]: Advancing to term 1
W20260924 12:47:35.508219 8798 leader_election.cc:334] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Timed out: connection negotiation to 127.5.77.193:41413 for RPC RequestConsensusVote timed out after 0.007s (ON_OUTBOUND_QUEUE)
I20260924 12:47:35.508404 8798 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:35.508730 8861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "4b04ce5d5bd546d880ced66f9b15decd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9"
I20260924 12:47:35.512331 9063 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 12:47:35.515214 8589 catalog_manager.cc:5495] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195). New cstate: current_term: 1 leader_uuid: "4b04ce5d5bd546d880ced66f9b15decd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } } }
I20260924 12:47:35.530323 9063 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:35.530892 9063 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:35.532202 8862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9" is_pre_election: true
I20260924 12:47:35.532209 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:35.532793 8862 raft_consensus.cc:2396] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0764fc96f6b048d6af98e1e628eef36b for term 1 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260924 12:47:35.533366 8666 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0764fc96f6b048d6af98e1e628eef36b; no voters: 34e93f57f57f41d49de37a374ea07bb9, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:35.534010 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:35.534216 9065 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 12:47:35.536293 9063 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:47:35.536556 8731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:47:35.537163 8798 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:35.537606 9063 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 12:47:35.592613 9081 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260924 12:47:35.592742 9081 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20260924 12:47:36.064388 8730 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260924 12:47:36.064476 8730 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20260924 12:47:36.065151 8862 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260924 12:47:36.065260 8862 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260924 12:47:36.425810 9087 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:36.426095 9087 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:36.427556 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:36.427752 9087 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:47:36.428300 8861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9" is_pre_election: true
I20260924 12:47:36.428956 8666 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0764fc96f6b048d6af98e1e628eef36b; no voters: 34e93f57f57f41d49de37a374ea07bb9, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:36.429364 9087 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:36.717593 9087 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:36.717808 9087 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:36.718667 9087 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:47:36.719105 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:36.719130 8861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9" is_pre_election: true
I20260924 12:47:36.719853 8666 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0764fc96f6b048d6af98e1e628eef36b; no voters: 34e93f57f57f41d49de37a374ea07bb9, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:36.720139 9087 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:36.720705 9090 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=0, attempt_no=0}) took 1134 ms. Trace:
I20260924 12:47:36.720829 9090 rpcz_store.cc:276] 0924 12:47:35.586326 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:35.586375 (+ 49us) service_pool.cc:227] Handling call
0924 12:47:36.720649 (+1134274us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:36.062596 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:47:36.062722 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:36.062729 (+ 7us) write_op.cc:435] Acquired schema lock
0924 12:47:36.062731 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:36.062849 (+ 118us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:36.062884 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:36.062888 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:36.429994 (+367106us) lock_manager.cc:599] Waited 366936us for lock on <redacted> (key hash 17251042410248443849)
0924 12:47:36.430007 (+ 13us) tablet.cc:699] Row locks acquired
0924 12:47:36.430010 (+ 3us) write_op.cc:260] PREPARE: finished
0924 12:47:36.430061 (+ 51us) write_op.cc:270] Start()
0924 12:47:36.498213 (+ 68152us) write_op.cc:276] Timestamp: P: 1790254056430020 usec, L: 0
0924 12:47:36.498219 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:47:36.498445 (+ 226us) log.cc:842] Serialized 345 byte log entry
0924 12:47:36.718490 (+220045us) op_driver.cc:444] REPLICATION: finished
0924 12:47:36.718638 (+ 148us) write_op.cc:301] APPLY: starting
0924 12:47:36.718697 (+ 59us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:36.718742 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:36.718744 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:36.718893 (+ 149us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:36.718928 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:36.718934 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:47:36.719190 (+ 256us) log.cc:842] Serialized 43 byte log entry
0924 12:47:36.719266 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:36.719296 (+ 30us) write_op.cc:454] Released schema lock
0924 12:47:36.719303 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":1433472,"child_traces":[["op",{"apply.queue_time_us":45,"mutex_wait_us":29,"num_ops":1,"prepare.queue_time_us":472377,"prepare.run_cpu_time_us":1012,"prepare.run_wall_time_us":435957,"replication_time_us":220218,"row_lock_wait_count":1,"row_lock_wait_us":366936}]]}
I20260924 12:47:37.044324 9088 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:37.044575 9088 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:37.045332 9088 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:47:37.046000 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:37.046476 8730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:47:37.047190 8798 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:37.454205 9095 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:37.454437 9095 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:37.455240 9095 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:47:37.455740 8861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9" is_pre_election: true
I20260924 12:47:37.455739 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:37.456454 8666 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0764fc96f6b048d6af98e1e628eef36b; no voters: 34e93f57f57f41d49de37a374ea07bb9, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:37.456988 9095 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:37.457396 9098 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=2, attempt_no=0}) took 1025 ms. Trace:
I20260924 12:47:37.457522 9098 rpcz_store.cc:276] 0924 12:47:36.431931 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:36.432010 (+ 79us) service_pool.cc:227] Handling call
0924 12:47:37.457335 (+1025325us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:36.498592 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:47:36.498720 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:36.498725 (+ 5us) write_op.cc:435] Acquired schema lock
0924 12:47:36.498727 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:36.498802 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:36.498812 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:36.498815 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:36.498862 (+ 47us) tablet.cc:699] Row locks acquired
0924 12:47:36.498863 (+ 1us) write_op.cc:260] PREPARE: finished
0924 12:47:36.498906 (+ 43us) write_op.cc:270] Start()
0924 12:47:37.268055 (+769149us) write_op.cc:276] Timestamp: P: 1790254056498861 usec, L: 0
0924 12:47:37.268061 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:47:37.268250 (+ 189us) log.cc:842] Serialized 354 byte log entry
0924 12:47:37.455726 (+187476us) op_driver.cc:444] REPLICATION: finished
0924 12:47:37.456517 (+ 791us) write_op.cc:301] APPLY: starting
0924 12:47:37.456536 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:37.456577 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:37.456674 (+ 97us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:37.456810 (+ 136us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:37.456853 (+ 43us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:37.456858 (+ 5us) write_op.cc:312] APPLY: finished
0924 12:47:37.456943 (+ 85us) log.cc:842] Serialized 22 byte log entry
0924 12:47:37.457218 (+ 275us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:37.457237 (+ 19us) write_op.cc:454] Released schema lock
0924 12:47:37.457244 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":725,"num_ops":1,"prepare.queue_time_us":66188,"prepare.run_cpu_time_us":681,"prepare.run_wall_time_us":769745,"raft.queue_time_us":46,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":170,"replication_time_us":187632,"thread_start_us":607,"threads_started":1}]]}
I20260924 12:47:38.046110 9088 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:38.107560 9095 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:38.107756 9095 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:38.108506 9095 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:47:38.108912 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:38.109057 8861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9" is_pre_election: true
I20260924 12:47:38.109721 8666 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0764fc96f6b048d6af98e1e628eef36b; no voters: 34e93f57f57f41d49de37a374ea07bb9, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:38.110354 9095 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:38.110682 9104 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=3, attempt_no=0}) took 1386 ms. Trace:
I20260924 12:47:38.110824 9104 rpcz_store.cc:276] 0924 12:47:36.723761 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:36.723828 (+ 67us) service_pool.cc:227] Handling call
0924 12:47:38.110551 (+1386723us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:37.268380 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:47:37.268501 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:37.268508 (+ 7us) write_op.cc:435] Acquired schema lock
0924 12:47:37.268510 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:37.268641 (+ 131us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:37.268651 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:37.268656 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:37.268712 (+ 56us) tablet.cc:699] Row locks acquired
0924 12:47:37.268715 (+ 3us) write_op.cc:260] PREPARE: finished
0924 12:47:37.268763 (+ 48us) write_op.cc:270] Start()
0924 12:47:37.557912 (+289149us) write_op.cc:276] Timestamp: P: 1790254057268744 usec, L: 0
0924 12:47:37.557918 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:47:37.558145 (+ 227us) log.cc:842] Serialized 354 byte log entry
0924 12:47:38.108896 (+550751us) op_driver.cc:444] REPLICATION: finished
0924 12:47:38.109816 (+ 920us) write_op.cc:301] APPLY: starting
0924 12:47:38.109834 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:38.109880 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:38.109917 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:38.110000 (+ 83us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:38.110037 (+ 37us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:38.110044 (+ 7us) write_op.cc:312] APPLY: finished
0924 12:47:38.110133 (+ 89us) log.cc:842] Serialized 22 byte log entry
0924 12:47:38.110409 (+ 276us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:38.110428 (+ 19us) write_op.cc:454] Released schema lock
0924 12:47:38.110435 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":854,"num_ops":1,"prepare.queue_time_us":543962,"prepare.run_cpu_time_us":786,"prepare.run_wall_time_us":289842,"replication_time_us":550947,"thread_start_us":742,"threads_started":1}]]}
I20260924 12:47:38.153906 9095 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:38.154098 9095 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:38.154887 9095 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:47:38.155371 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:38.155401 8861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9" is_pre_election: true
I20260924 12:47:38.156214 8664 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0764fc96f6b048d6af98e1e628eef36b; no voters: 34e93f57f57f41d49de37a374ea07bb9, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:38.156577 9095 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:38.665302 9088 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:38.665494 9088 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:38.666262 9088 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:47:38.666800 8730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:47:38.667084 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:38.667562 8795 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:38.985056 9088 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:39.062896 9088 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:39.063082 9088 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:39.063817 9088 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:47:39.064299 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:39.064471 8731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:47:39.065320 8798 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:39.065660 9088 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:39.449903 9088 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:39.450165 9088 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:39.450981 9088 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:47:39.451520 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:39.451726 8731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:47:39.451879 9109 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=6, attempt_no=0}) took 1294 ms. Trace:
I20260924 12:47:39.452000 9109 rpcz_store.cc:276] 0924 12:47:38.157244 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:38.157308 (+ 64us) service_pool.cc:227] Handling call
0924 12:47:39.451804 (+1294496us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:38.829020 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:47:38.829154 (+ 134us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:38.829177 (+ 23us) write_op.cc:435] Acquired schema lock
0924 12:47:38.829179 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:38.829300 (+ 121us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:38.829312 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:38.829317 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:38.829376 (+ 59us) tablet.cc:699] Row locks acquired
0924 12:47:38.829378 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:38.829426 (+ 48us) write_op.cc:270] Start()
0924 12:47:39.106533 (+277107us) write_op.cc:276] Timestamp: P: 1790254058829402 usec, L: 0
0924 12:47:39.106539 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:47:39.106736 (+ 197us) log.cc:842] Serialized 354 byte log entry
0924 12:47:39.451090 (+344354us) op_driver.cc:444] REPLICATION: finished
0924 12:47:39.451229 (+ 139us) write_op.cc:301] APPLY: starting
0924 12:47:39.451252 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:39.451298 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:39.451324 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:39.451401 (+ 77us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:39.451446 (+ 45us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:39.451452 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:47:39.451565 (+ 113us) log.cc:842] Serialized 22 byte log entry
0924 12:47:39.451670 (+ 105us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:39.451691 (+ 21us) write_op.cc:454] Released schema lock
0924 12:47:39.451698 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":78,"num_ops":1,"prepare.queue_time_us":671206,"prepare.run_cpu_time_us":755,"prepare.run_wall_time_us":277786,"replication_time_us":344516}]]}
I20260924 12:47:39.452664 8798 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:39.452906 9088 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:39.812141 9108 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:39.812383 9108 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:39.813212 9108 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:47:39.813714 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:39.814518 8861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9" is_pre_election: true
I20260924 12:47:39.815279 8666 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0764fc96f6b048d6af98e1e628eef36b; no voters: 34e93f57f57f41d49de37a374ea07bb9, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:40.366000 9088 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:40.366191 9088 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:40.366963 9088 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:47:40.367409 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:40.367533 8731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:47:40.368439 8795 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:40.368692 9117 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=7, attempt_no=0}) took 1302 ms. Trace:
I20260924 12:47:40.368839 9117 rpcz_store.cc:276] 0924 12:47:39.066714 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:39.066794 (+ 80us) service_pool.cc:227] Handling call
0924 12:47:40.368642 (+1301848us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:39.106837 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:47:39.106931 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:39.106936 (+ 5us) write_op.cc:435] Acquired schema lock
0924 12:47:39.106937 (+ 1us) tablet.cc:660] Decoding operations
0924 12:47:39.107018 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:39.107028 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:39.107031 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:39.107075 (+ 44us) tablet.cc:699] Row locks acquired
0924 12:47:39.107077 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:39.107114 (+ 37us) write_op.cc:270] Start()
0924 12:47:39.504250 (+397136us) write_op.cc:276] Timestamp: P: 1790254059107074 usec, L: 0
0924 12:47:39.504254 (+ 4us) op_driver.cc:331] REPLICATION: starting
0924 12:47:39.504488 (+ 234us) log.cc:842] Serialized 354 byte log entry
0924 12:47:40.367657 (+863169us) op_driver.cc:444] REPLICATION: finished
0924 12:47:40.368077 (+ 420us) write_op.cc:301] APPLY: starting
0924 12:47:40.368096 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:40.368139 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:40.368208 (+ 69us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:40.368296 (+ 88us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:40.368325 (+ 29us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:40.368329 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:47:40.368416 (+ 87us) log.cc:842] Serialized 22 byte log entry
0924 12:47:40.368513 (+ 97us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:40.368533 (+ 20us) write_op.cc:454] Released schema lock
0924 12:47:40.368540 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":358,"num_ops":1,"prepare.queue_time_us":39592,"prepare.run_cpu_time_us":670,"prepare.run_wall_time_us":397724,"replication_time_us":863378,"thread_start_us":245,"threads_started":1}]]}
I20260924 12:47:40.373116 9108 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:40.770939 9088 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:40.772712 9117 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=8, attempt_no=0}) took 1318 ms. Trace:
I20260924 12:47:40.772836 9117 rpcz_store.cc:276] 0924 12:47:39.454124 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:39.454195 (+ 71us) service_pool.cc:227] Handling call
0924 12:47:40.772641 (+1318446us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:39.504611 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:47:39.504719 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:39.504724 (+ 5us) write_op.cc:435] Acquired schema lock
0924 12:47:39.504725 (+ 1us) tablet.cc:660] Decoding operations
0924 12:47:39.504802 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:39.504812 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:39.504815 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:39.504860 (+ 45us) tablet.cc:699] Row locks acquired
0924 12:47:39.504861 (+ 1us) write_op.cc:260] PREPARE: finished
0924 12:47:39.504905 (+ 44us) write_op.cc:270] Start()
0924 12:47:39.813039 (+308134us) write_op.cc:276] Timestamp: P: 1790254059504865 usec, L: 0
0924 12:47:39.813044 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:47:39.813409 (+ 365us) log.cc:842] Serialized 354 byte log entry
0924 12:47:40.771986 (+958577us) op_driver.cc:444] REPLICATION: finished
0924 12:47:40.772108 (+ 122us) write_op.cc:301] APPLY: starting
0924 12:47:40.772126 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:40.772189 (+ 63us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:40.772221 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:40.772323 (+ 102us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:40.772360 (+ 37us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:40.772366 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:47:40.772448 (+ 82us) log.cc:842] Serialized 22 byte log entry
0924 12:47:40.772527 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:40.772544 (+ 17us) write_op.cc:454] Released schema lock
0924 12:47:40.772551 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":51,"num_ops":1,"prepare.queue_time_us":49919,"prepare.run_cpu_time_us":748,"prepare.run_wall_time_us":308868,"replication_time_us":958914,"spinlock_wait_cycles":320256}]]}
I20260924 12:47:41.274730 9108 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:41.274938 9108 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:41.275641 9108 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:47:41.276098 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:41.276170 8861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9" is_pre_election: true
I20260924 12:47:41.276714 8664 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0764fc96f6b048d6af98e1e628eef36b; no voters: 34e93f57f57f41d49de37a374ea07bb9, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:41.738693 9108 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:41.741104 9124 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=9, attempt_no=0}) took 1370 ms. Trace:
I20260924 12:47:41.741326 9124 rpcz_store.cc:276] 0924 12:47:40.370359 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:40.370418 (+ 59us) service_pool.cc:227] Handling call
0924 12:47:41.741039 (+1370621us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:40.371156 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:47:40.371292 (+ 136us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:40.371295 (+ 3us) write_op.cc:435] Acquired schema lock
0924 12:47:40.371297 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:40.371384 (+ 87us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:40.371393 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:40.371395 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:40.371446 (+ 51us) tablet.cc:699] Row locks acquired
0924 12:47:40.371448 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:40.371487 (+ 39us) write_op.cc:270] Start()
0924 12:47:40.999639 (+628152us) write_op.cc:276] Timestamp: P: 1790254060371464 usec, L: 0
0924 12:47:40.999643 (+ 4us) op_driver.cc:331] REPLICATION: starting
0924 12:47:40.999850 (+ 207us) log.cc:842] Serialized 354 byte log entry
0924 12:47:41.739919 (+740069us) op_driver.cc:444] REPLICATION: finished
0924 12:47:41.740388 (+ 469us) write_op.cc:301] APPLY: starting
0924 12:47:41.740406 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:41.740451 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:41.740494 (+ 43us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:41.740665 (+ 171us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:41.740705 (+ 40us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:41.740723 (+ 18us) write_op.cc:312] APPLY: finished
0924 12:47:41.740828 (+ 105us) log.cc:842] Serialized 22 byte log entry
0924 12:47:41.740907 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:41.740930 (+ 23us) write_op.cc:454] Released schema lock
0924 12:47:41.740937 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":407,"num_ops":1,"prepare.queue_time_us":296,"prepare.run_cpu_time_us":714,"prepare.run_wall_time_us":628769,"replication_time_us":740252,"thread_start_us":473,"threads_started":2}]]}
I20260924 12:47:41.875700 9088 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:41.875905 9088 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:41.876616 9088 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:47:41.877063 8730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:47:41.877261 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:41.877847 8795 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:42.213380 9088 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:42.666011 9131 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:42.666270 9131 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:42.667207 9131 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:47:42.667683 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:42.667759 8861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9" is_pre_election: true
I20260924 12:47:42.668459 8666 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0764fc96f6b048d6af98e1e628eef36b; no voters: 34e93f57f57f41d49de37a374ea07bb9, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:42.669008 9131 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:43.037112 9088 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:43.037330 9088 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:43.038182 9088 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:47:43.038573 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:43.038615 8731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:47:43.039229 8798 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:43.039532 9088 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:43.789314 9137 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:43.789594 9137 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:43.790357 9137 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:47:43.790904 8861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9" is_pre_election: true
I20260924 12:47:43.790977 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:43.791678 8666 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0764fc96f6b048d6af98e1e628eef36b; no voters: 34e93f57f57f41d49de37a374ea07bb9, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:43.793020 9141 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:43.793257 9140 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=12, attempt_no=0}) took 2049 ms. Trace:
I20260924 12:47:43.793403 9140 rpcz_store.cc:276] 0924 12:47:41.743443 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:41.743506 (+ 63us) service_pool.cc:227] Handling call
0924 12:47:43.793186 (+2049680us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:42.582022 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:47:42.582153 (+ 131us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:42.582159 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:47:42.582162 (+ 3us) tablet.cc:660] Decoding operations
0924 12:47:42.582264 (+ 102us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:42.582296 (+ 32us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:42.582301 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:42.582357 (+ 56us) tablet.cc:699] Row locks acquired
0924 12:47:42.582360 (+ 3us) write_op.cc:260] PREPARE: finished
0924 12:47:42.582409 (+ 49us) write_op.cc:270] Start()
0924 12:47:43.462569 (+880160us) write_op.cc:276] Timestamp: P: 1790254062582381 usec, L: 0
0924 12:47:43.462573 (+ 4us) op_driver.cc:331] REPLICATION: starting
0924 12:47:43.462921 (+ 348us) log.cc:842] Serialized 354 byte log entry
0924 12:47:43.790739 (+327818us) op_driver.cc:444] REPLICATION: finished
0924 12:47:43.791343 (+ 604us) write_op.cc:301] APPLY: starting
0924 12:47:43.791364 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:43.791409 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:43.791465 (+ 56us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:43.792247 (+ 782us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:43.792416 (+ 169us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:43.792423 (+ 7us) write_op.cc:312] APPLY: finished
0924 12:47:43.792669 (+ 246us) log.cc:842] Serialized 22 byte log entry
0924 12:47:43.792990 (+ 321us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:43.793030 (+ 40us) write_op.cc:454] Released schema lock
0924 12:47:43.793040 (+ 10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":540,"num_ops":1,"prepare.queue_time_us":838120,"prepare.run_cpu_time_us":892,"prepare.run_wall_time_us":881000,"raft.queue_time_us":38,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":248,"replication_time_us":328135,"spinlock_wait_cycles":263808,"thread_start_us":407,"threads_started":1}]]}
I20260924 12:47:43.998171 9088 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:43.998358 9088 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:43.999181 9088 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:47:43.999616 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:44.000022 8730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:47:44.000638 8798 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:44.334398 9088 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:44.336593 9145 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=13, attempt_no=0}) took 1666 ms. Trace:
I20260924 12:47:44.336786 9145 rpcz_store.cc:276] 0924 12:47:42.670226 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:42.670287 (+ 61us) service_pool.cc:227] Handling call
0924 12:47:44.336520 (+1666233us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:43.463058 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:47:43.463208 (+ 150us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:43.463213 (+ 5us) write_op.cc:435] Acquired schema lock
0924 12:47:43.463214 (+ 1us) tablet.cc:660] Decoding operations
0924 12:47:43.463293 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:43.463305 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:43.463309 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:43.793060 (+329751us) lock_manager.cc:599] Waited 329664us for lock on <redacted> (key hash 11785448468237383701)
0924 12:47:43.793071 (+ 11us) tablet.cc:699] Row locks acquired
0924 12:47:43.793074 (+ 3us) write_op.cc:260] PREPARE: finished
0924 12:47:43.793257 (+ 183us) write_op.cc:270] Start()
0924 12:47:43.953415 (+160158us) write_op.cc:276] Timestamp: P: 1790254063793174 usec, L: 0
0924 12:47:43.953421 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:47:43.953675 (+ 254us) log.cc:842] Serialized 354 byte log entry
0924 12:47:44.335555 (+381880us) op_driver.cc:444] REPLICATION: finished
0924 12:47:44.335968 (+ 413us) write_op.cc:301] APPLY: starting
0924 12:47:44.336016 (+ 48us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:44.336063 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:44.336065 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:44.336141 (+ 76us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:44.336173 (+ 32us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:44.336178 (+ 5us) write_op.cc:312] APPLY: finished
0924 12:47:44.336298 (+ 120us) log.cc:842] Serialized 43 byte log entry
0924 12:47:44.336369 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:44.336388 (+ 19us) write_op.cc:454] Released schema lock
0924 12:47:44.336396 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":358,"num_ops":1,"prepare.queue_time_us":792320,"prepare.run_cpu_time_us":994,"prepare.run_wall_time_us":490726,"raft.queue_time_us":42,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":382104,"row_lock_wait_count":1,"row_lock_wait_us":329664,"spinlock_wait_cycles":233088,"thread_start_us":255,"threads_started":1}]]}
I20260924 12:47:44.536466 9141 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:44.536712 9141 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:44.537443 9141 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:47:44.538017 8861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9" is_pre_election: true
I20260924 12:47:44.538017 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:44.538687 8664 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0764fc96f6b048d6af98e1e628eef36b; no voters: 34e93f57f57f41d49de37a374ea07bb9, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:45.062520 9088 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:45.062743 9088 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:45.063421 9088 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:47:45.063894 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:45.064024 8730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:47:45.065315 8795 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:45.065546 9151 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=14, attempt_no=0}) took 1270 ms. Trace:
I20260924 12:47:45.065755 9151 rpcz_store.cc:276] 0924 12:47:43.795340 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:43.795408 (+ 68us) service_pool.cc:227] Handling call
0924 12:47:45.065482 (+1270074us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:43.953816 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:47:43.953972 (+ 156us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:43.953979 (+ 7us) write_op.cc:435] Acquired schema lock
0924 12:47:43.953981 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:43.954108 (+ 127us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:43.954121 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:43.954127 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:43.954183 (+ 56us) tablet.cc:699] Row locks acquired
0924 12:47:43.954185 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:43.954234 (+ 49us) write_op.cc:270] Start()
0924 12:47:44.395371 (+441137us) write_op.cc:276] Timestamp: P: 1790254063954203 usec, L: 0
0924 12:47:44.395375 (+ 4us) op_driver.cc:331] REPLICATION: starting
0924 12:47:44.395551 (+ 176us) log.cc:842] Serialized 354 byte log entry
0924 12:47:45.063953 (+668402us) op_driver.cc:444] REPLICATION: finished
0924 12:47:45.064437 (+ 484us) write_op.cc:301] APPLY: starting
0924 12:47:45.064513 (+ 76us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:45.064558 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:45.064999 (+ 441us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:45.065095 (+ 96us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:45.065130 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:45.065136 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:47:45.065277 (+ 141us) log.cc:842] Serialized 22 byte log entry
0924 12:47:45.065354 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:45.065374 (+ 20us) write_op.cc:454] Released schema lock
0924 12:47:45.065381 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":414,"num_ops":1,"prepare.queue_time_us":157941,"prepare.run_cpu_time_us":764,"prepare.run_wall_time_us":441819,"raft.queue_time_us":41,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":228,"replication_time_us":668552,"thread_start_us":293,"threads_started":1}]]}
I20260924 12:47:45.375731 9088 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:45.377429 9151 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=15, attempt_no=0}) took 1038 ms. Trace:
I20260924 12:47:45.377549 9151 rpcz_store.cc:276] 0924 12:47:44.338636 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:44.338754 (+ 118us) service_pool.cc:227] Handling call
0924 12:47:45.377369 (+1038615us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:44.395674 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:47:44.395802 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:44.395808 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:47:44.395810 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:44.395910 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:44.395922 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:44.395927 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:44.395976 (+ 49us) tablet.cc:699] Row locks acquired
0924 12:47:44.395978 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:44.396018 (+ 40us) write_op.cc:270] Start()
0924 12:47:45.015171 (+619153us) write_op.cc:276] Timestamp: P: 1790254064395958 usec, L: 0
0924 12:47:45.015177 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:47:45.015425 (+ 248us) log.cc:842] Serialized 354 byte log entry
0924 12:47:45.376737 (+361312us) op_driver.cc:444] REPLICATION: finished
0924 12:47:45.376851 (+ 114us) write_op.cc:301] APPLY: starting
0924 12:47:45.376870 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:45.376918 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:45.376942 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:45.377009 (+ 67us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:45.377041 (+ 32us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:45.377045 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:47:45.377126 (+ 81us) log.cc:842] Serialized 22 byte log entry
0924 12:47:45.377231 (+ 105us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:45.377251 (+ 20us) write_op.cc:454] Released schema lock
0924 12:47:45.377269 (+ 18us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":35,"num_ops":1,"prepare.queue_time_us":56451,"prepare.run_cpu_time_us":760,"prepare.run_wall_time_us":619820,"replication_time_us":361530}]]}
I20260924 12:47:45.447965 9141 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:45.504962 9088 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:45.505180 9088 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:45.505877 9088 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:47:45.506433 8730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:47:45.506477 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:45.507186 8795 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:45.507565 9088 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:46.771842 9141 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:46.772094 8731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.5.77.195:39535 (request call id 1901) took 1322 ms. Trace:
I20260924 12:47:46.772137 9141 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:46.772948 9141 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:47:46.773515 8861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9" is_pre_election: true
I20260924 12:47:46.773517 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:46.772248 8731 rpcz_store.cc:276] 0924 12:47:45.449435 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:45.449501 (+ 66us) service_pool.cc:227] Handling call
0924 12:47:45.449684 (+ 183us) raft_consensus.cc:1457] Updating replica for 2 ops
0924 12:47:45.449868 (+ 184us) raft_consensus.cc:1520] Early marking committed up to index 16
0924 12:47:45.450189 (+ 321us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0924 12:47:45.450362 (+ 173us) write_op.cc:270] Start()
0924 12:47:46.116526 (+666164us) write_op.cc:276] Timestamp: P: 1790254064395958 usec, L: 0
0924 12:47:46.117149 (+ 623us) write_op.cc:270] Start()
0924 12:47:46.770335 (+653186us) write_op.cc:276] Timestamp: P: 1790254065068362 usec, L: 0
0924 12:47:46.771016 (+ 681us) log.cc:842] Serialized 708 byte log entry
0924 12:47:46.771393 (+ 377us) raft_consensus.cc:1631] Marking committed up to 17
0924 12:47:46.771732 (+ 339us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 12:47:46.771777 (+ 45us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 12:47:46.771929 (+ 152us) raft_consensus.cc:1671] finished
0924 12:47:46.771932 (+ 3us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 12:47:46.771990 (+ 58us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:46.117069 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:47:46.117176 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:46.117181 (+ 5us) write_op.cc:435] Acquired schema lock
0924 12:47:46.117183 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:46.117326 (+ 143us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:46.117348 (+ 22us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:46.117353 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:46.117420 (+ 67us) tablet.cc:699] Row locks acquired
0924 12:47:46.117422 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:46.771435 (+654013us) op_driver.cc:444] REPLICATION: finished
0924 12:47:46.771833 (+ 398us) write_op.cc:301] APPLY: starting
0924 12:47:46.771850 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:46.771891 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:46.771893 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:46.771963 (+ 70us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:46.771996 (+ 33us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:46.772036 (+ 40us) write_op.cc:312] APPLY: finished
0924 12:47:46.772125 (+ 89us) log.cc:842] Serialized 22 byte log entry
0924 12:47:46.772274 (+ 149us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:46.772291 (+ 17us) write_op.cc:454] Released schema lock
0924 12:47:46.772298 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0924 12:47:46.772380 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:46.772384 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0924 12:47:46.770895 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:47:46.771000 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:46.771005 (+ 5us) write_op.cc:435] Acquired schema lock
0924 12:47:46.771006 (+ 1us) tablet.cc:660] Decoding operations
0924 12:47:46.771124 (+ 118us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:46.771141 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:46.771146 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:46.771218 (+ 72us) tablet.cc:699] Row locks acquired
0924 12:47:46.771220 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":296,"threads_started":1,"wal-append.queue_time_us":369,"child_traces":[["op",{"apply.queue_time_us":336,"apply.run_cpu_time_us":551,"apply.run_wall_time_us":613,"num_ops":1,"partition_lock_wait_us":10,"prepare.queue_time_us":403,"prepare.run_cpu_time_us":462,"prepare.run_wall_time_us":462,"replication_time_us":1321058,"thread_start_us":514,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":416,"prepare.run_cpu_time_us":422,"prepare.run_wall_time_us":422,"thread_start_us":247,"threads_started":1}]]}
I20260924 12:47:46.774178 8664 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0764fc96f6b048d6af98e1e628eef36b; no voters: 34e93f57f57f41d49de37a374ea07bb9, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:46.774504 9141 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:46.830076 9088 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:46.830252 9088 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:46.831049 9088 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:47:46.831501 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:46.831594 8730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:47:46.832187 8798 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:46.832527 9088 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:46.833791 9164 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=17, attempt_no=0}) took 1454 ms. Trace:
I20260924 12:47:46.833906 9164 rpcz_store.cc:276] 0924 12:47:45.379436 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:45.379495 (+ 59us) service_pool.cc:227] Handling call
0924 12:47:46.833703 (+1454208us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:45.380075 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:47:45.380191 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:45.380197 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:47:45.380200 (+ 3us) tablet.cc:660] Decoding operations
0924 12:47:45.380307 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:45.380318 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:45.380323 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:45.380377 (+ 54us) tablet.cc:699] Row locks acquired
0924 12:47:45.380380 (+ 3us) write_op.cc:260] PREPARE: finished
0924 12:47:45.380419 (+ 39us) write_op.cc:270] Start()
0924 12:47:45.945562 (+565143us) write_op.cc:276] Timestamp: P: 1790254065380368 usec, L: 0
0924 12:47:45.945566 (+ 4us) op_driver.cc:331] REPLICATION: starting
0924 12:47:45.945770 (+ 204us) log.cc:842] Serialized 354 byte log entry
0924 12:47:46.831746 (+885976us) op_driver.cc:444] REPLICATION: finished
0924 12:47:46.832928 (+ 1182us) write_op.cc:301] APPLY: starting
0924 12:47:46.832946 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:46.832982 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:46.833022 (+ 40us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:46.833100 (+ 78us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:46.833132 (+ 32us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:46.833138 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:47:46.833279 (+ 141us) log.cc:842] Serialized 43 byte log entry
0924 12:47:46.833542 (+ 263us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:46.833561 (+ 19us) write_op.cc:454] Released schema lock
0924 12:47:46.833569 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1111,"num_ops":1,"prepare.queue_time_us":49,"prepare.run_cpu_time_us":731,"prepare.run_wall_time_us":565798,"raft.queue_time_us":30,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":172,"replication_time_us":886148,"spinlock_wait_cycles":25088,"thread_start_us":1034,"threads_started":1}]]}
I20260924 12:47:47.189097 9088 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:47.189340 9088 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:47.190176 9088 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:47:47.190727 9164 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=18, attempt_no=0}) took 1682 ms. Trace:
I20260924 12:47:47.191030 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:47.191447 8730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:47:47.190871 9164 rpcz_store.cc:276] 0924 12:47:45.508446 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:45.508515 (+ 69us) service_pool.cc:227] Handling call
0924 12:47:47.190660 (+1682145us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:45.945921 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:47:45.946071 (+ 150us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:45.946078 (+ 7us) write_op.cc:435] Acquired schema lock
0924 12:47:45.946080 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:45.946225 (+ 145us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:45.946237 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:45.946242 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:45.946302 (+ 60us) tablet.cc:699] Row locks acquired
0924 12:47:45.946304 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:45.946354 (+ 50us) write_op.cc:270] Start()
0924 12:47:46.597504 (+651150us) write_op.cc:276] Timestamp: P: 1790254065946297 usec, L: 0
0924 12:47:46.597509 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:47:46.597772 (+ 263us) log.cc:842] Serialized 354 byte log entry
0924 12:47:47.190021 (+592249us) op_driver.cc:444] REPLICATION: finished
0924 12:47:47.190109 (+ 88us) write_op.cc:301] APPLY: starting
0924 12:47:47.190129 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:47.190174 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:47.190215 (+ 41us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:47.190293 (+ 78us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:47.190331 (+ 38us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:47.190337 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:47:47.190435 (+ 98us) log.cc:842] Serialized 22 byte log entry
0924 12:47:47.190538 (+ 103us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:47.190557 (+ 19us) write_op.cc:454] Released schema lock
0924 12:47:47.190565 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":36,"num_ops":1,"prepare.queue_time_us":436915,"prepare.run_cpu_time_us":873,"prepare.run_wall_time_us":651934,"replication_time_us":592485}]]}
I20260924 12:47:47.192186 8798 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:47.192447 9088 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:47.542255 9088 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:47.542497 9088 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:47.543196 9088 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:47:47.543767 8730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:47:47.543846 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:47.545029 8795 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:47.545310 9088 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:47.609282 9141 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:47.609503 9141 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:47.610220 9141 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:47:47.610695 8862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9" is_pre_election: true
I20260924 12:47:47.610709 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:47.611395 8664 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0764fc96f6b048d6af98e1e628eef36b; no voters: 34e93f57f57f41d49de37a374ea07bb9, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:48.078075 9141 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:48.500967 9088 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:48.501184 9088 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:48.501925 9088 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:47:48.502486 8730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:47:48.502418 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:48.503279 8798 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:48.503814 9170 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=20, attempt_no=0}) took 1310 ms. Trace:
I20260924 12:47:48.503938 9170 rpcz_store.cc:276] 0924 12:47:47.193247 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:47.193328 (+ 81us) service_pool.cc:227] Handling call
0924 12:47:48.503770 (+1310442us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:47.288208 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:47:47.288370 (+ 162us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:47.288376 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:47:47.288377 (+ 1us) tablet.cc:660] Decoding operations
0924 12:47:47.288454 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:47.288463 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:47.288466 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:47.288512 (+ 46us) tablet.cc:699] Row locks acquired
0924 12:47:47.288513 (+ 1us) write_op.cc:260] PREPARE: finished
0924 12:47:47.288552 (+ 39us) write_op.cc:270] Start()
0924 12:47:47.895688 (+607136us) write_op.cc:276] Timestamp: P: 1790254067288516 usec, L: 0
0924 12:47:47.895693 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:47:47.895908 (+ 215us) log.cc:842] Serialized 354 byte log entry
0924 12:47:48.502453 (+606545us) op_driver.cc:444] REPLICATION: finished
0924 12:47:48.503199 (+ 746us) write_op.cc:301] APPLY: starting
0924 12:47:48.503216 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:48.503258 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:48.503283 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:48.503382 (+ 99us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:48.503418 (+ 36us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:48.503424 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:47:48.503512 (+ 88us) log.cc:842] Serialized 22 byte log entry
0924 12:47:48.503614 (+ 102us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:48.503637 (+ 23us) write_op.cc:454] Released schema lock
0924 12:47:48.503646 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":653,"num_ops":1,"prepare.queue_time_us":94364,"prepare.run_cpu_time_us":725,"prepare.run_wall_time_us":607786,"raft.queue_time_us":31,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":208,"replication_time_us":606731,"thread_start_us":556,"threads_started":1}]]}
I20260924 12:47:49.054987 9141 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:49.055241 9141 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:49.055897 9141 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:47:49.056417 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:49.056506 8862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9" is_pre_election: true
I20260924 12:47:49.057098 8666 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0764fc96f6b048d6af98e1e628eef36b; no voters: 34e93f57f57f41d49de37a374ea07bb9, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:49.061688 9088 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:49.064014 9177 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=21, attempt_no=0}) took 1518 ms. Trace:
I20260924 12:47:49.064137 9177 rpcz_store.cc:276] 0924 12:47:47.545673 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:47.545738 (+ 65us) service_pool.cc:227] Handling call
0924 12:47:49.063957 (+1518219us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:47.896017 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:47:47.896142 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:47.896149 (+ 7us) write_op.cc:435] Acquired schema lock
0924 12:47:47.896151 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:47.896247 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:47.896259 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:47.896264 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:47.896335 (+ 71us) tablet.cc:699] Row locks acquired
0924 12:47:47.896337 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:47.896387 (+ 50us) write_op.cc:270] Start()
0924 12:47:48.295539 (+399152us) write_op.cc:276] Timestamp: P: 1790254067896336 usec, L: 0
0924 12:47:48.295544 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:47:48.295777 (+ 233us) log.cc:842] Serialized 354 byte log entry
0924 12:47:49.062953 (+767176us) op_driver.cc:444] REPLICATION: finished
0924 12:47:49.063432 (+ 479us) write_op.cc:301] APPLY: starting
0924 12:47:49.063450 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:49.063492 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:49.063512 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:49.063593 (+ 81us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:49.063625 (+ 32us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:49.063630 (+ 5us) write_op.cc:312] APPLY: finished
0924 12:47:49.063749 (+ 119us) log.cc:842] Serialized 43 byte log entry
0924 12:47:49.063813 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:49.063830 (+ 17us) write_op.cc:454] Released schema lock
0924 12:47:49.063838 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":417,"num_ops":1,"prepare.queue_time_us":349845,"prepare.run_cpu_time_us":760,"prepare.run_wall_time_us":399824,"replication_time_us":767383,"thread_start_us":297,"threads_started":1}]]}
I20260924 12:47:49.495517 9141 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:50.172638 9141 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:50.172857 9141 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:50.173516 9141 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:47:50.174093 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:50.174171 8862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9" is_pre_election: true
I20260924 12:47:50.174767 8664 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0764fc96f6b048d6af98e1e628eef36b; no voters: 34e93f57f57f41d49de37a374ea07bb9, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:50.175022 9141 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:50.176499 9182 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=22, attempt_no=0}) took 1670 ms. Trace:
I20260924 12:47:50.176656 9182 rpcz_store.cc:276] 0924 12:47:48.505805 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:48.505888 (+ 83us) service_pool.cc:227] Handling call
0924 12:47:50.176442 (+1670554us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:48.506364 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:47:48.506488 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:48.506493 (+ 5us) write_op.cc:435] Acquired schema lock
0924 12:47:48.506494 (+ 1us) tablet.cc:660] Decoding operations
0924 12:47:48.506612 (+ 118us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:48.506621 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:48.506625 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:48.506676 (+ 51us) tablet.cc:699] Row locks acquired
0924 12:47:48.506678 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:48.506714 (+ 36us) write_op.cc:270] Start()
0924 12:47:49.445880 (+939166us) write_op.cc:276] Timestamp: P: 1790254068506659 usec, L: 0
0924 12:47:49.445887 (+ 7us) op_driver.cc:331] REPLICATION: starting
0924 12:47:49.446104 (+ 217us) log.cc:842] Serialized 354 byte log entry
0924 12:47:50.175410 (+729306us) op_driver.cc:444] REPLICATION: finished
0924 12:47:50.175872 (+ 462us) write_op.cc:301] APPLY: starting
0924 12:47:50.175890 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:50.175951 (+ 61us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:50.175983 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:50.176063 (+ 80us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:50.176098 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:50.176103 (+ 5us) write_op.cc:312] APPLY: finished
0924 12:47:50.176216 (+ 113us) log.cc:842] Serialized 43 byte log entry
0924 12:47:50.176292 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:50.176321 (+ 29us) write_op.cc:454] Released schema lock
0924 12:47:50.176328 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":401,"num_ops":1,"prepare.queue_time_us":68,"prepare.run_cpu_time_us":765,"prepare.run_wall_time_us":939836,"raft.queue_time_us":47,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":223,"replication_time_us":729498,"thread_start_us":312,"threads_started":1}]]}
I20260924 12:47:50.422406 9088 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:50.422616 9088 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:50.423296 9088 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:47:50.423699 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:50.423719 8731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:47:50.424333 8795 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:50.425002 9088 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:51.254383 9187 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:51.254638 9187 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:51.255505 9187 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:47:51.256120 8862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9" is_pre_election: true
I20260924 12:47:51.256126 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:51.256819 8666 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0764fc96f6b048d6af98e1e628eef36b; no voters: 34e93f57f57f41d49de37a374ea07bb9, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:51.257083 9187 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:51.258492 9192 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=23, attempt_no=0}) took 2192 ms. Trace:
I20260924 12:47:51.258600 9192 rpcz_store.cc:276] 0924 12:47:49.065935 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:49.066000 (+ 65us) service_pool.cc:227] Handling call
0924 12:47:51.258420 (+2192420us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:49.446219 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:47:49.446325 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:49.446330 (+ 5us) write_op.cc:435] Acquired schema lock
0924 12:47:49.446332 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:49.446409 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:49.446419 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:49.446422 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:50.176490 (+730068us) lock_manager.cc:599] Waited 729954us for lock on <redacted> (key hash 17251042410248443849)
0924 12:47:50.176502 (+ 12us) tablet.cc:699] Row locks acquired
0924 12:47:50.176505 (+ 3us) write_op.cc:260] PREPARE: finished
0924 12:47:50.176544 (+ 39us) write_op.cc:270] Start()
0924 12:47:50.745679 (+569135us) write_op.cc:276] Timestamp: P: 1790254070176520 usec, L: 0
0924 12:47:50.745683 (+ 4us) op_driver.cc:331] REPLICATION: starting
0924 12:47:50.745858 (+ 175us) log.cc:842] Serialized 354 byte log entry
0924 12:47:51.256797 (+510939us) op_driver.cc:444] REPLICATION: finished
0924 12:47:51.257486 (+ 689us) write_op.cc:301] APPLY: starting
0924 12:47:51.257532 (+ 46us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:51.257577 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:51.257580 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:51.257691 (+ 111us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:51.257738 (+ 47us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:51.257744 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:47:51.257855 (+ 111us) log.cc:842] Serialized 43 byte log entry
0924 12:47:51.258202 (+ 347us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:51.258232 (+ 30us) write_op.cc:454] Released schema lock
0924 12:47:51.258239 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":621,"num_ops":1,"prepare.queue_time_us":379710,"prepare.run_cpu_time_us":725,"prepare.run_wall_time_us":1299726,"raft.queue_time_us":76,"raft.run_cpu_time_us":551,"raft.run_wall_time_us":625,"replication_time_us":511087,"row_lock_wait_count":1,"row_lock_wait_us":729954,"spinlock_wait_cycles":148864,"thread_start_us":1088,"threads_started":1}]]}
I20260924 12:47:51.376358 9088 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:51.376582 9088 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:51.377509 9088 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:47:51.377851 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:51.377864 8731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:47:51.378419 8798 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:51.379134 9088 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:52.285140 9088 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:52.285337 9088 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:52.286043 9088 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:47:52.286660 8731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:47:52.286676 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:52.287295 8798 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:52.287561 9088 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:52.288625 9199 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=24, attempt_no=0}) took 2110 ms. Trace:
I20260924 12:47:52.288761 9199 rpcz_store.cc:276] 0924 12:47:50.178558 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:50.178619 (+ 61us) service_pool.cc:227] Handling call
0924 12:47:52.288559 (+2109940us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:50.746016 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:47:50.746209 (+ 193us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:50.746217 (+ 8us) write_op.cc:435] Acquired schema lock
0924 12:47:50.746219 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:50.746437 (+ 218us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:50.746473 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:50.746479 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:51.258250 (+511771us) lock_manager.cc:599] Waited 511689us for lock on <redacted> (key hash 17251042410248443849)
0924 12:47:51.258262 (+ 12us) tablet.cc:699] Row locks acquired
0924 12:47:51.258265 (+ 3us) write_op.cc:260] PREPARE: finished
0924 12:47:51.258415 (+ 150us) write_op.cc:270] Start()
0924 12:47:51.693510 (+435095us) write_op.cc:276] Timestamp: P: 1790254071258382 usec, L: 0
0924 12:47:51.693515 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:47:51.693698 (+ 183us) log.cc:842] Serialized 354 byte log entry
0924 12:47:52.286929 (+593231us) op_driver.cc:444] REPLICATION: finished
0924 12:47:52.288006 (+ 1077us) write_op.cc:301] APPLY: starting
0924 12:47:52.288049 (+ 43us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:52.288090 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:52.288093 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:52.288193 (+ 100us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:52.288224 (+ 31us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:52.288231 (+ 7us) write_op.cc:312] APPLY: finished
0924 12:47:52.288356 (+ 125us) log.cc:842] Serialized 43 byte log entry
0924 12:47:52.288418 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:52.288436 (+ 18us) write_op.cc:454] Released schema lock
0924 12:47:52.288443 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1008,"num_ops":1,"prepare.queue_time_us":566979,"prepare.run_cpu_time_us":990,"prepare.run_wall_time_us":947775,"raft.queue_time_us":38,"raft.run_cpu_time_us":283,"raft.run_wall_time_us":520,"replication_time_us":593388,"row_lock_wait_count":1,"row_lock_wait_us":511689,"spinlock_wait_cycles":210688,"thread_start_us":885,"threads_started":1}]]}
I20260924 12:47:52.492681 9187 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:52.492868 9187 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:52.493575 9187 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:47:52.494064 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:52.494124 8862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9" is_pre_election: true
I20260924 12:47:52.494767 8666 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0764fc96f6b048d6af98e1e628eef36b; no voters: 34e93f57f57f41d49de37a374ea07bb9, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:52.544080 9088 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:52.544289 9088 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:52.545046 9088 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:47:52.545497 8731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:47:52.545531 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:52.546011 9199 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=25, attempt_no=0}) took 1285 ms. Trace:
I20260924 12:47:52.546047 8798 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:52.546305 9088 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:52.546124 9199 rpcz_store.cc:276] 0924 12:47:51.260412 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:51.260471 (+ 59us) service_pool.cc:227] Handling call
0924 12:47:52.545957 (+1285486us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:51.693830 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:47:51.693948 (+ 118us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:51.693954 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:47:51.693956 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:51.694058 (+ 102us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:51.694069 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:51.694074 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:51.694129 (+ 55us) tablet.cc:699] Row locks acquired
0924 12:47:51.694131 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:51.694168 (+ 37us) write_op.cc:270] Start()
0924 12:47:51.936307 (+242139us) write_op.cc:276] Timestamp: P: 1790254071694137 usec, L: 0
0924 12:47:51.936312 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:47:51.936600 (+ 288us) log.cc:842] Serialized 354 byte log entry
0924 12:47:52.545250 (+608650us) op_driver.cc:444] REPLICATION: finished
0924 12:47:52.545368 (+ 118us) write_op.cc:301] APPLY: starting
0924 12:47:52.545386 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:52.545434 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:52.545465 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:52.545634 (+ 169us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:52.545667 (+ 33us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:52.545673 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:47:52.545756 (+ 83us) log.cc:842] Serialized 22 byte log entry
0924 12:47:52.545847 (+ 91us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:52.545865 (+ 18us) write_op.cc:454] Released schema lock
0924 12:47:52.545872 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":67,"num_ops":1,"prepare.queue_time_us":432963,"prepare.run_cpu_time_us":792,"prepare.run_wall_time_us":242849,"replication_time_us":608907}]]}
I20260924 12:47:53.211977 9088 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:53.212186 9088 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:53.212877 9088 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:47:53.213411 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:53.213557 8731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:47:53.214178 8798 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:53.214752 9088 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:53.489739 9187 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:53.975322 9088 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:53.975533 9088 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:53.976291 9088 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:47:53.976740 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:53.976855 8731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:47:53.977572 8798 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:53.977840 9214 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=27, attempt_no=0}) took 1429 ms. Trace:
I20260924 12:47:53.977962 9214 rpcz_store.cc:276] 0924 12:47:52.547940 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:52.548027 (+ 87us) service_pool.cc:227] Handling call
0924 12:47:53.977778 (+1429751us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:52.548472 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:47:52.548562 (+ 90us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:52.548566 (+ 4us) write_op.cc:435] Acquired schema lock
0924 12:47:52.548568 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:52.548637 (+ 69us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:52.548645 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:52.548648 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:52.548688 (+ 40us) tablet.cc:699] Row locks acquired
0924 12:47:52.548690 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:52.548721 (+ 31us) write_op.cc:270] Start()
0924 12:47:53.265868 (+717147us) write_op.cc:276] Timestamp: P: 1790254072548668 usec, L: 0
0924 12:47:53.265874 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:47:53.266082 (+ 208us) log.cc:842] Serialized 354 byte log entry
0924 12:47:53.976717 (+710635us) op_driver.cc:444] REPLICATION: finished
0924 12:47:53.977257 (+ 540us) write_op.cc:301] APPLY: starting
0924 12:47:53.977276 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:53.977319 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:53.977349 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:53.977429 (+ 80us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:53.977464 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:53.977470 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:47:53.977558 (+ 88us) log.cc:842] Serialized 22 byte log entry
0924 12:47:53.977643 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:53.977662 (+ 19us) write_op.cc:454] Released schema lock
0924 12:47:53.977669 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":469,"num_ops":1,"prepare.queue_time_us":53,"prepare.run_cpu_time_us":612,"prepare.run_wall_time_us":717681,"replication_time_us":710816,"thread_start_us":350,"threads_started":1}]]}
I20260924 12:47:54.078310 9187 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:54.078531 9187 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:54.079300 9187 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:47:54.079695 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:54.079864 8862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9" is_pre_election: true
I20260924 12:47:54.080446 8666 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0764fc96f6b048d6af98e1e628eef36b; no voters: 34e93f57f57f41d49de37a374ea07bb9, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:54.210259 9088 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:54.250348 9187 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:54.927001 9088 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:54.927199 9088 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:54.927909 9088 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:47:54.928514 8731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:47:54.928524 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:54.929283 8798 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:54.929672 9088 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:55.031499 9187 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:55.031687 9187 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:55.032451 9187 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:47:55.033051 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:55.033794 8861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9" is_pre_election: true
I20260924 12:47:55.034312 8666 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0764fc96f6b048d6af98e1e628eef36b; no voters: 34e93f57f57f41d49de37a374ea07bb9, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:55.034621 9187 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:55.198482 9187 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:55.198675 9187 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:55.199419 9187 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:47:55.199764 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:55.200419 8861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9" is_pre_election: true
I20260924 12:47:55.200942 8666 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0764fc96f6b048d6af98e1e628eef36b; no voters: 34e93f57f57f41d49de37a374ea07bb9, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:55.201337 9187 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:55.395290 9187 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:55.395490 9187 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:55.396265 9187 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:47:55.396766 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:55.396937 8862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9" is_pre_election: true
I20260924 12:47:55.397768 9219 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=30, attempt_no=0}) took 1184 ms. Trace:
I20260924 12:47:55.397890 9219 rpcz_store.cc:276] 0924 12:47:54.213607 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:54.213670 (+ 63us) service_pool.cc:227] Handling call
0924 12:47:55.397701 (+1184031us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:54.521514 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:47:54.521668 (+ 154us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:54.521675 (+ 7us) write_op.cc:435] Acquired schema lock
0924 12:47:54.521678 (+ 3us) tablet.cc:660] Decoding operations
0924 12:47:54.521799 (+ 121us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:54.521811 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:54.521815 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:54.521875 (+ 60us) tablet.cc:699] Row locks acquired
0924 12:47:54.521877 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:54.521923 (+ 46us) write_op.cc:270] Start()
0924 12:47:55.267097 (+745174us) write_op.cc:276] Timestamp: P: 1790254074521874 usec, L: 0
0924 12:47:55.267103 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:47:55.267326 (+ 223us) log.cc:842] Serialized 354 byte log entry
0924 12:47:55.396610 (+129284us) op_driver.cc:444] REPLICATION: finished
0924 12:47:55.397147 (+ 537us) write_op.cc:301] APPLY: starting
0924 12:47:55.397166 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:55.397229 (+ 63us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:55.397270 (+ 41us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:55.397347 (+ 77us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:55.397383 (+ 36us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:55.397388 (+ 5us) write_op.cc:312] APPLY: finished
0924 12:47:55.397495 (+ 107us) log.cc:842] Serialized 43 byte log entry
0924 12:47:55.397558 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:55.397577 (+ 19us) write_op.cc:454] Released schema lock
0924 12:47:55.397585 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":477,"num_ops":1,"prepare.queue_time_us":307384,"prepare.run_cpu_time_us":1187,"prepare.run_wall_time_us":746242,"raft.queue_time_us":370,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":7,"replication_time_us":129461,"thread_start_us":306,"threads_started":1}]]}
I20260924 12:47:55.398445 8664 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0764fc96f6b048d6af98e1e628eef36b; no voters: 34e93f57f57f41d49de37a374ea07bb9, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:55.398783 9187 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:55.571755 9088 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:55.571964 9088 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:55.572748 9088 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:47:55.573041 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:55.573366 8730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:47:55.574258 8798 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:55.574522 9088 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:55.980438 9088 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:55.980645 9088 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:55.981465 9088 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:47:55.981917 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:55.982117 8730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:47:55.982846 8798 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:55.983228 9226 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=31, attempt_no=0}) took 1052 ms. Trace:
I20260924 12:47:55.983268 9088 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:55.983350 9226 rpcz_store.cc:276] 0924 12:47:54.930941 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:54.931008 (+ 67us) service_pool.cc:227] Handling call
0924 12:47:55.983159 (+1052151us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:55.267809 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:47:55.267961 (+ 152us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:55.267967 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:47:55.267970 (+ 3us) tablet.cc:660] Decoding operations
0924 12:47:55.268067 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:55.268098 (+ 31us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:55.268101 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:55.268148 (+ 47us) tablet.cc:699] Row locks acquired
0924 12:47:55.268150 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:55.268192 (+ 42us) write_op.cc:270] Start()
0924 12:47:55.787335 (+519143us) write_op.cc:276] Timestamp: P: 1790254075268159 usec, L: 0
0924 12:47:55.787339 (+ 4us) op_driver.cc:331] REPLICATION: starting
0924 12:47:55.787528 (+ 189us) log.cc:842] Serialized 354 byte log entry
0924 12:47:55.981483 (+193955us) op_driver.cc:444] REPLICATION: finished
0924 12:47:55.982443 (+ 960us) write_op.cc:301] APPLY: starting
0924 12:47:55.982487 (+ 44us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:55.982532 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:55.982562 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:55.982641 (+ 79us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:55.982674 (+ 33us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:55.982679 (+ 5us) write_op.cc:312] APPLY: finished
0924 12:47:55.982952 (+ 273us) log.cc:842] Serialized 43 byte log entry
0924 12:47:55.983016 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:55.983032 (+ 16us) write_op.cc:454] Released schema lock
0924 12:47:55.983039 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":889,"mutex_wait_us":17,"num_ops":1,"prepare.queue_time_us":336366,"prepare.run_cpu_time_us":794,"prepare.run_wall_time_us":519860,"raft.queue_time_us":81,"raft.run_cpu_time_us":477,"raft.run_wall_time_us":477,"replication_time_us":194115,"spinlock_wait_cycles":139136,"thread_start_us":1627,"threads_started":1}]]}
I20260924 12:47:56.471866 9187 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:56.472074 9187 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:56.472714 9187 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:47:56.473171 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:56.473186 8862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9" is_pre_election: true
I20260924 12:47:56.473810 8666 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0764fc96f6b048d6af98e1e628eef36b; no voters: 34e93f57f57f41d49de37a374ea07bb9, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:56.995065 9187 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:56.997964 9233 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=32, attempt_no=0}) took 1597 ms. Trace:
I20260924 12:47:56.998104 9233 rpcz_store.cc:276] 0924 12:47:55.400021 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:55.400087 (+ 66us) service_pool.cc:227] Handling call
0924 12:47:56.997899 (+1597812us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:55.787711 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:47:55.787837 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:55.787843 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:47:55.787845 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:55.787941 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:55.787952 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:55.787957 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:55.788016 (+ 59us) tablet.cc:699] Row locks acquired
0924 12:47:55.788018 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:55.788067 (+ 49us) write_op.cc:270] Start()
0924 12:47:56.237224 (+449157us) write_op.cc:276] Timestamp: P: 1790254075788030 usec, L: 0
0924 12:47:56.237228 (+ 4us) op_driver.cc:331] REPLICATION: starting
0924 12:47:56.237415 (+ 187us) log.cc:842] Serialized 354 byte log entry
0924 12:47:56.996543 (+759128us) op_driver.cc:444] REPLICATION: finished
0924 12:47:56.997176 (+ 633us) write_op.cc:301] APPLY: starting
0924 12:47:56.997195 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:56.997238 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:56.997271 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:56.997392 (+ 121us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:56.997430 (+ 38us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:56.997436 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:47:56.997553 (+ 117us) log.cc:842] Serialized 22 byte log entry
0924 12:47:56.997771 (+ 218us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:56.997792 (+ 21us) write_op.cc:454] Released schema lock
0924 12:47:56.997799 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":571,"num_ops":1,"prepare.queue_time_us":387155,"prepare.run_cpu_time_us":728,"prepare.run_wall_time_us":449791,"replication_time_us":759284,"thread_start_us":462,"threads_started":1}]]}
I20260924 12:47:57.159322 9187 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:57.159498 9187 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:57.160244 9187 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:47:57.160696 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:57.160781 8862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9" is_pre_election: true
I20260924 12:47:57.161216 9233 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=33, attempt_no=0}) took 1176 ms. Trace:
I20260924 12:47:57.161471 8666 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0764fc96f6b048d6af98e1e628eef36b; no voters: 34e93f57f57f41d49de37a374ea07bb9, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:57.161331 9233 rpcz_store.cc:276] 0924 12:47:55.985167 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:55.985223 (+ 56us) service_pool.cc:227] Handling call
0924 12:47:57.161156 (+1175933us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:56.237522 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:47:56.237696 (+ 174us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:56.237702 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:47:56.237704 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:56.237812 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:56.237824 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:56.237830 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:56.237878 (+ 48us) tablet.cc:699] Row locks acquired
0924 12:47:56.237880 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:56.237919 (+ 39us) write_op.cc:270] Start()
0924 12:47:57.144068 (+906149us) write_op.cc:276] Timestamp: P: 1790254076237897 usec, L: 0
0924 12:47:57.144074 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:47:57.144322 (+ 248us) log.cc:842] Serialized 354 byte log entry
0924 12:47:57.160464 (+ 16142us) op_driver.cc:444] REPLICATION: finished
0924 12:47:57.160582 (+ 118us) write_op.cc:301] APPLY: starting
0924 12:47:57.160600 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:57.160644 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:57.160667 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:57.160751 (+ 84us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:57.160798 (+ 47us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:57.160804 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:47:57.160913 (+ 109us) log.cc:842] Serialized 43 byte log entry
0924 12:47:57.160988 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:57.161006 (+ 18us) write_op.cc:454] Released schema lock
0924 12:47:57.161013 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":60,"num_ops":1,"prepare.queue_time_us":251899,"prepare.run_cpu_time_us":811,"prepare.run_wall_time_us":906878,"replication_time_us":16361}]]}
I20260924 12:47:57.162303 9187 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:57.195066 9088 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:57.195254 9088 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:57.196014 9088 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:47:57.196645 8731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:47:57.196739 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:57.197220 8798 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:57.197467 9088 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:58.056272 9088 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:58.056468 9088 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:58.057298 9088 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:47:58.057749 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:58.058055 8730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:47:58.058678 8798 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:58.344393 9238 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:58.344660 9238 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:58.346078 8862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9" is_pre_election: true
I20260924 12:47:58.346082 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:58.345463 9238 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:47:58.346967 8666 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0764fc96f6b048d6af98e1e628eef36b; no voters: 34e93f57f57f41d49de37a374ea07bb9, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:58.347566 9238 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:58.347826 9244 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=34, attempt_no=0}) took 1347 ms. Trace:
I20260924 12:47:58.347991 9244 rpcz_store.cc:276] 0924 12:47:56.999854 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:56.999914 (+ 60us) service_pool.cc:227] Handling call
0924 12:47:58.347639 (+1347725us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:57.144448 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:47:57.144563 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:57.144568 (+ 5us) write_op.cc:435] Acquired schema lock
0924 12:47:57.144569 (+ 1us) tablet.cc:660] Decoding operations
0924 12:47:57.144658 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:57.144669 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:57.144674 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:57.144731 (+ 57us) tablet.cc:699] Row locks acquired
0924 12:47:57.144733 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:57.144794 (+ 61us) write_op.cc:270] Start()
0924 12:47:57.881940 (+737146us) write_op.cc:276] Timestamp: P: 1790254077144726 usec, L: 0
0924 12:47:57.881946 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:47:57.882177 (+ 231us) log.cc:842] Serialized 354 byte log entry
0924 12:47:58.346290 (+464113us) op_driver.cc:444] REPLICATION: finished
0924 12:47:58.347052 (+ 762us) write_op.cc:301] APPLY: starting
0924 12:47:58.347073 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:58.347118 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:58.347145 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:58.347223 (+ 78us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:58.347258 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:58.347264 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:47:58.347387 (+ 123us) log.cc:842] Serialized 43 byte log entry
0924 12:47:58.347458 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:58.347480 (+ 22us) write_op.cc:454] Released schema lock
0924 12:47:58.347487 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":681,"num_ops":1,"prepare.queue_time_us":144113,"prepare.run_cpu_time_us":748,"prepare.run_wall_time_us":737813,"replication_time_us":464315,"thread_start_us":573,"threads_started":1}]]}
I20260924 12:47:58.635915 9088 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:59.234129 9238 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:59.234380 9238 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:59.235050 9238 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:47:59.235543 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:59.235713 8862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9" is_pre_election: true
I20260924 12:47:59.236389 8666 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0764fc96f6b048d6af98e1e628eef36b; no voters: 34e93f57f57f41d49de37a374ea07bb9, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:59.237035 9238 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:59.237659 9251 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=35, attempt_no=0}) took 2074 ms. Trace:
I20260924 12:47:59.237766 9251 rpcz_store.cc:276] 0924 12:47:57.163160 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:57.163222 (+ 62us) service_pool.cc:227] Handling call
0924 12:47:59.237622 (+2074400us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:57.882320 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:47:57.882498 (+ 178us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:57.882505 (+ 7us) write_op.cc:435] Acquired schema lock
0924 12:47:57.882507 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:57.882596 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:57.882606 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:57.882609 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:57.882655 (+ 46us) tablet.cc:699] Row locks acquired
0924 12:47:57.882656 (+ 1us) write_op.cc:260] PREPARE: finished
0924 12:47:57.882705 (+ 49us) write_op.cc:270] Start()
0924 12:47:58.431843 (+549138us) write_op.cc:276] Timestamp: P: 1790254077882665 usec, L: 0
0924 12:47:58.431847 (+ 4us) op_driver.cc:331] REPLICATION: starting
0924 12:47:58.432054 (+ 207us) log.cc:842] Serialized 354 byte log entry
0924 12:47:59.235663 (+803609us) op_driver.cc:444] REPLICATION: finished
0924 12:47:59.236825 (+ 1162us) write_op.cc:301] APPLY: starting
0924 12:47:59.236841 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:59.236879 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:59.236902 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:59.236983 (+ 81us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:59.237016 (+ 33us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:59.237022 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:47:59.237164 (+ 142us) log.cc:842] Serialized 43 byte log entry
0924 12:47:59.237454 (+ 290us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:59.237474 (+ 20us) write_op.cc:454] Released schema lock
0924 12:47:59.237482 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1104,"num_ops":1,"prepare.queue_time_us":718651,"prepare.run_cpu_time_us":773,"prepare.run_wall_time_us":549837,"raft.queue_time_us":36,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":233,"replication_time_us":803788,"thread_start_us":972,"threads_started":1}]]}
I20260924 12:47:59.276959 9088 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:59.277161 9088 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:59.277812 9088 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:47:59.278138 8731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:47:59.279183 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:59.279815 8795 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:59.280081 9088 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:59.302719 9088 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:59.302901 9088 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:59.303588 9088 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:47:59.303936 8730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:47:59.304524 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:59.305064 8795 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:59.305315 9088 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:59.402344 9238 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:59.402555 9238 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:59.403256 9238 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:47:59.403600 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:59.403757 8861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9" is_pre_election: true
I20260924 12:47:59.404356 8666 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0764fc96f6b048d6af98e1e628eef36b; no voters: 34e93f57f57f41d49de37a374ea07bb9, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:59.404632 9238 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:59.862133 9088 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:47:59.862540 9088 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:47:59.863379 9088 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:47:59.864002 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:47:59.864009 8731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:47:59.864720 8798 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:47:59.865422 9088 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:47:59.865411 9257 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=36, attempt_no=0}) took 1515 ms. Trace:
I20260924 12:47:59.865548 9257 rpcz_store.cc:276] 0924 12:47:58.350084 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:58.350157 (+ 73us) service_pool.cc:227] Handling call
0924 12:47:59.865347 (+1515190us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:58.432195 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:47:58.432329 (+ 134us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:58.432336 (+ 7us) write_op.cc:435] Acquired schema lock
0924 12:47:58.432338 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:58.432452 (+ 114us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:58.432463 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:58.432466 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:58.432515 (+ 49us) tablet.cc:699] Row locks acquired
0924 12:47:58.432517 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:58.432559 (+ 42us) write_op.cc:270] Start()
0924 12:47:59.423689 (+991130us) write_op.cc:276] Timestamp: P: 1790254078432521 usec, L: 0
0924 12:47:59.423695 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:47:59.423935 (+ 240us) log.cc:842] Serialized 354 byte log entry
0924 12:47:59.863273 (+439338us) op_driver.cc:444] REPLICATION: finished
0924 12:47:59.864489 (+ 1216us) write_op.cc:301] APPLY: starting
0924 12:47:59.864512 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0924 12:47:59.864572 (+ 60us) tablet.cc:1368] finished BulkCheckPresence
0924 12:47:59.864601 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:47:59.864708 (+ 107us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:47:59.864745 (+ 37us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:47:59.864752 (+ 7us) write_op.cc:312] APPLY: finished
0924 12:47:59.864878 (+ 126us) log.cc:842] Serialized 43 byte log entry
0924 12:47:59.865184 (+ 306us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:47:59.865206 (+ 22us) write_op.cc:454] Released schema lock
0924 12:47:59.865215 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":530,"num_ops":1,"prepare.queue_time_us":81484,"prepare.run_cpu_time_us":777,"prepare.run_wall_time_us":991830,"raft.queue_time_us":41,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":238,"replication_time_us":439546,"thread_start_us":510,"threads_started":1}]]}
I20260924 12:48:00.219128 9238 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:48:00.219323 9238 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:48:00.219995 9238 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:48:00.220525 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:48:00.220625 8862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9" is_pre_election: true
I20260924 12:48:00.221248 8666 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0764fc96f6b048d6af98e1e628eef36b; no voters: 34e93f57f57f41d49de37a374ea07bb9, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:48:00.221541 9238 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:48:00.507484 9088 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:48:00.507722 9088 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:48:00.508472 9088 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:48:00.509019 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:48:00.509037 8731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:48:00.509734 8798 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:48:00.510046 9088 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:48:00.510380 9263 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=37, attempt_no=0}) took 1270 ms. Trace:
I20260924 12:48:00.510510 9263 rpcz_store.cc:276] 0924 12:47:59.239595 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:59.239651 (+ 56us) service_pool.cc:227] Handling call
0924 12:48:00.510327 (+1270676us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:47:59.424061 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:47:59.424189 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:47:59.424195 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:47:59.424197 (+ 2us) tablet.cc:660] Decoding operations
0924 12:47:59.424313 (+ 116us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:47:59.424325 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:47:59.424329 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:47:59.424386 (+ 57us) tablet.cc:699] Row locks acquired
0924 12:47:59.424388 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:47:59.424433 (+ 45us) write_op.cc:270] Start()
0924 12:48:00.048581 (+624148us) write_op.cc:276] Timestamp: P: 1790254079424408 usec, L: 0
0924 12:48:00.048587 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:48:00.048805 (+ 218us) log.cc:842] Serialized 354 byte log entry
0924 12:48:00.509014 (+460209us) op_driver.cc:444] REPLICATION: finished
0924 12:48:00.509699 (+ 685us) write_op.cc:301] APPLY: starting
0924 12:48:00.509718 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0924 12:48:00.509774 (+ 56us) tablet.cc:1368] finished BulkCheckPresence
0924 12:48:00.509853 (+ 79us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:48:00.509939 (+ 86us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:48:00.509972 (+ 33us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:48:00.509978 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:48:00.510101 (+ 123us) log.cc:842] Serialized 43 byte log entry
0924 12:48:00.510179 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:48:00.510198 (+ 19us) write_op.cc:454] Released schema lock
0924 12:48:00.510206 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":623,"num_ops":1,"prepare.queue_time_us":184027,"prepare.run_cpu_time_us":753,"prepare.run_wall_time_us":624818,"replication_time_us":460396,"thread_start_us":327,"threads_started":1}]]}
I20260924 12:48:00.993767 9088 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:48:00.993983 9088 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:48:00.994865 9088 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:48:00.995358 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:48:00.995631 8731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:48:00.995720 9263 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=38, attempt_no=0}) took 1128 ms. Trace:
I20260924 12:48:00.995848 9263 rpcz_store.cc:276] 0924 12:47:59.867484 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:47:59.867550 (+ 66us) service_pool.cc:227] Handling call
0924 12:48:00.995649 (+1128099us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:48:00.048919 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:48:00.049037 (+ 118us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:48:00.049043 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:48:00.049045 (+ 2us) tablet.cc:660] Decoding operations
0924 12:48:00.049150 (+ 105us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:48:00.049162 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:48:00.049167 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:48:00.049226 (+ 59us) tablet.cc:699] Row locks acquired
0924 12:48:00.049228 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:48:00.049285 (+ 57us) write_op.cc:270] Start()
0924 12:48:00.797448 (+748163us) write_op.cc:276] Timestamp: P: 1790254080049255 usec, L: 0
0924 12:48:00.797454 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:48:00.797690 (+ 236us) log.cc:842] Serialized 354 byte log entry
0924 12:48:00.994862 (+197172us) op_driver.cc:444] REPLICATION: finished
0924 12:48:00.994990 (+ 128us) write_op.cc:301] APPLY: starting
0924 12:48:00.995017 (+ 27us) tablet.cc:1365] starting BulkCheckPresence
0924 12:48:00.995063 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0924 12:48:00.995091 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:48:00.995177 (+ 86us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:48:00.995214 (+ 37us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:48:00.995220 (+ 6us) write_op.cc:312] APPLY: finished
0924 12:48:00.995360 (+ 140us) log.cc:842] Serialized 43 byte log entry
0924 12:48:00.995435 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:48:00.995455 (+ 20us) write_op.cc:454] Released schema lock
0924 12:48:00.995464 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":69,"num_ops":1,"prepare.queue_time_us":180819,"prepare.run_cpu_time_us":772,"prepare.run_wall_time_us":748850,"replication_time_us":197372}]]}
I20260924 12:48:00.996227 8798 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:48:00.996940 9088 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:48:01.064584 9238 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:48:01.064810 9238 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:48:01.065591 9238 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:48:01.066016 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:48:01.066118 8861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9" is_pre_election: true
I20260924 12:48:01.066787 8666 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0764fc96f6b048d6af98e1e628eef36b; no voters: 34e93f57f57f41d49de37a374ea07bb9, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:48:01.986557 9238 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:48:01.995469 9238 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:48:01.995669 9238 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:48:01.996338 9238 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:48:01.996376 9271 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=39, attempt_no=0}) took 1484 ms. Trace:
I20260924 12:48:01.996572 9271 rpcz_store.cc:276] 0924 12:48:00.512343 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:48:00.512400 (+ 57us) service_pool.cc:227] Handling call
0924 12:48:01.996322 (+1483922us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:48:00.797814 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:48:00.797940 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:48:00.797946 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:48:00.797948 (+ 2us) tablet.cc:660] Decoding operations
0924 12:48:00.798048 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:48:00.798074 (+ 26us) write_op.cc:641] Partition lock acquired for write op
0924 12:48:00.798079 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:48:00.798136 (+ 57us) tablet.cc:699] Row locks acquired
0924 12:48:00.798138 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:48:00.798185 (+ 47us) write_op.cc:270] Start()
0924 12:48:01.780352 (+982167us) write_op.cc:276] Timestamp: P: 1790254080798122 usec, L: 0
0924 12:48:01.780359 (+ 7us) op_driver.cc:331] REPLICATION: starting
0924 12:48:01.780609 (+ 250us) log.cc:842] Serialized 354 byte log entry
0924 12:48:01.995224 (+214615us) op_driver.cc:444] REPLICATION: finished
0924 12:48:01.995708 (+ 484us) write_op.cc:301] APPLY: starting
0924 12:48:01.995725 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0924 12:48:01.995778 (+ 53us) tablet.cc:1368] finished BulkCheckPresence
0924 12:48:01.995810 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:48:01.995920 (+ 110us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:48:01.995951 (+ 31us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:48:01.995956 (+ 5us) write_op.cc:312] APPLY: finished
0924 12:48:01.996083 (+ 127us) log.cc:842] Serialized 43 byte log entry
0924 12:48:01.996152 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:48:01.996173 (+ 21us) write_op.cc:454] Released schema lock
0924 12:48:01.996180 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":428,"num_ops":1,"prepare.queue_time_us":285019,"prepare.run_cpu_time_us":817,"prepare.run_wall_time_us":982890,"raft.queue_time_us":33,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":178,"replication_time_us":214841,"thread_start_us":319,"threads_started":1}]]}
I20260924 12:48:01.997126 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:48:01.997292 8862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9" is_pre_election: true
I20260924 12:48:01.997949 8664 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0764fc96f6b048d6af98e1e628eef36b; no voters: 34e93f57f57f41d49de37a374ea07bb9, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:48:01.998201 9238 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:48:02.186209 9268 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:48:02.186509 9268 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:48:02.187301 9268 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:48:02.187741 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:48:02.187866 8731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:48:02.188448 8798 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:48:02.259601 9268 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:48:02.261296 9271 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=40, attempt_no=0}) took 1263 ms. Trace:
I20260924 12:48:02.261451 9271 rpcz_store.cc:276] 0924 12:48:00.998026 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:48:00.998098 (+ 72us) service_pool.cc:227] Handling call
0924 12:48:02.261219 (+1263121us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:48:01.780739 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:48:01.780869 (+ 130us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:48:01.780877 (+ 8us) write_op.cc:435] Acquired schema lock
0924 12:48:01.780879 (+ 2us) tablet.cc:660] Decoding operations
0924 12:48:01.780993 (+ 114us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:48:01.781005 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:48:01.781011 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:48:01.781087 (+ 76us) tablet.cc:699] Row locks acquired
0924 12:48:01.781090 (+ 3us) write_op.cc:260] PREPARE: finished
0924 12:48:01.781149 (+ 59us) write_op.cc:270] Start()
0924 12:48:02.032289 (+251140us) write_op.cc:276] Timestamp: P: 1790254081781096 usec, L: 0
0924 12:48:02.032294 (+ 5us) op_driver.cc:331] REPLICATION: starting
0924 12:48:02.032479 (+ 185us) log.cc:842] Serialized 354 byte log entry
0924 12:48:02.260584 (+228105us) op_driver.cc:444] REPLICATION: finished
0924 12:48:02.260730 (+ 146us) write_op.cc:301] APPLY: starting
0924 12:48:02.260765 (+ 35us) tablet.cc:1365] starting BulkCheckPresence
0924 12:48:02.260807 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0924 12:48:02.260828 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:48:02.260893 (+ 65us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:48:02.260920 (+ 27us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:48:02.260924 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:48:02.260997 (+ 73us) log.cc:842] Serialized 22 byte log entry
0924 12:48:02.261095 (+ 98us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:48:02.261116 (+ 21us) write_op.cc:454] Released schema lock
0924 12:48:02.261123 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":93,"num_ops":1,"prepare.queue_time_us":782094,"prepare.run_cpu_time_us":765,"prepare.run_wall_time_us":251825,"raft.queue_time_us":43,"raft.run_cpu_time_us":46,"raft.run_wall_time_us":47,"replication_time_us":228260,"spinlock_wait_cycles":78592}]]}
I20260924 12:48:02.382117 9268 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:48:02.382318 9268 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:48:02.383180 9268 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:48:02.383638 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:48:02.383647 8731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:48:02.384469 8795 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:48:02.384939 9268 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:48:02.884351 9238 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:48:02.884559 9238 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:48:02.885231 9238 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:48:02.885671 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:48:02.885774 8862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9" is_pre_election: true
I20260924 12:48:02.886368 8666 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0764fc96f6b048d6af98e1e628eef36b; no voters: 34e93f57f57f41d49de37a374ea07bb9, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:48:03.024554 9268 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:48:03.024786 9268 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:48:03.025581 9268 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:48:03.026197 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 44 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:48:03.026198 8731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 44 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:48:03.026985 8795 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:48:03.707592 9268 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:48:03.709966 9284 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=43, attempt_no=0}) took 1323 ms. Trace:
I20260924 12:48:03.710150 9284 rpcz_store.cc:276] 0924 12:48:02.386043 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:48:02.386148 (+ 105us) service_pool.cc:227] Handling call
0924 12:48:03.709913 (+1323765us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:48:02.481383 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:48:02.481500 (+ 117us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:48:02.481506 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:48:02.481509 (+ 3us) tablet.cc:660] Decoding operations
0924 12:48:02.481661 (+ 152us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:48:02.481673 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:48:02.481678 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:48:02.481736 (+ 58us) tablet.cc:699] Row locks acquired
0924 12:48:02.481739 (+ 3us) write_op.cc:260] PREPARE: finished
0924 12:48:02.481782 (+ 43us) write_op.cc:270] Start()
0924 12:48:02.852966 (+371184us) write_op.cc:276] Timestamp: P: 1790254082481752 usec, L: 0
0924 12:48:02.852972 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:48:02.853230 (+ 258us) log.cc:842] Serialized 354 byte log entry
0924 12:48:03.709022 (+855792us) op_driver.cc:444] REPLICATION: finished
0924 12:48:03.709471 (+ 449us) write_op.cc:301] APPLY: starting
0924 12:48:03.709493 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0924 12:48:03.709535 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0924 12:48:03.709564 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:48:03.709635 (+ 71us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:48:03.709662 (+ 27us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:48:03.709666 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:48:03.709747 (+ 81us) log.cc:842] Serialized 22 byte log entry
0924 12:48:03.709815 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:48:03.709831 (+ 16us) write_op.cc:454] Released schema lock
0924 12:48:03.709836 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":388,"num_ops":1,"prepare.queue_time_us":94730,"prepare.run_cpu_time_us":849,"prepare.run_wall_time_us":371933,"replication_time_us":856021,"thread_start_us":293,"threads_started":1}]]}
I20260924 12:48:03.963238 9268 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:48:03.963439 9268 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:48:03.964228 9268 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:48:03.964855 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:48:03.964869 8731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:48:03.965649 8795 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:48:03.965936 9268 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:48:04.673367 9268 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:48:04.673621 9268 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:48:04.674365 9268 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:48:04.674999 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:48:04.674985 8731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:48:04.675673 8795 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:48:04.676355 9268 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:48:04.808627 9238 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:48:04.809065 8730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.5.77.195:39535 (request call id 3457) took 1923 ms. Trace:
I20260924 12:48:04.809216 8730 rpcz_store.cc:276] 0924 12:48:02.885797 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:48:02.885858 (+ 61us) service_pool.cc:227] Handling call
0924 12:48:02.886058 (+ 200us) raft_consensus.cc:1457] Updating replica for 3 ops
0924 12:48:02.886218 (+ 160us) raft_consensus.cc:1520] Early marking committed up to index 42
0924 12:48:02.886539 (+ 321us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0924 12:48:02.886660 (+ 121us) write_op.cc:270] Start()
0924 12:48:03.790826 (+904166us) write_op.cc:276] Timestamp: P: 1790254082032914 usec, L: 0
0924 12:48:03.791481 (+ 655us) write_op.cc:270] Start()
0924 12:48:04.631644 (+840163us) write_op.cc:276] Timestamp: P: 1790254082313873 usec, L: 0
0924 12:48:04.632217 (+ 573us) write_op.cc:270] Start()
0924 12:48:04.807391 (+175174us) write_op.cc:276] Timestamp: P: 1790254082481752 usec, L: 0
0924 12:48:04.807757 (+ 366us) log.cc:842] Serialized 1062 byte log entry
0924 12:48:04.808151 (+ 394us) raft_consensus.cc:1631] Marking committed up to 43
0924 12:48:04.808495 (+ 344us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 12:48:04.808535 (+ 40us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 12:48:04.808899 (+ 364us) raft_consensus.cc:1671] finished
0924 12:48:04.808901 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 12:48:04.809007 (+ 106us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:48:03.791335 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:48:03.791477 (+ 142us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:48:03.791483 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:48:03.791484 (+ 1us) tablet.cc:660] Decoding operations
0924 12:48:03.791569 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:48:03.791584 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0924 12:48:03.791587 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:48:03.791637 (+ 50us) tablet.cc:699] Row locks acquired
0924 12:48:03.791639 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:48:04.808195 (+1016556us) op_driver.cc:444] REPLICATION: finished
0924 12:48:04.808616 (+ 421us) write_op.cc:301] APPLY: starting
0924 12:48:04.808632 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0924 12:48:04.808676 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0924 12:48:04.808678 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:48:04.808786 (+ 108us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:48:04.808819 (+ 33us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:48:04.808884 (+ 65us) write_op.cc:312] APPLY: finished
0924 12:48:04.808985 (+ 101us) log.cc:842] Serialized 43 byte log entry
0924 12:48:04.809051 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:48:04.809065 (+ 14us) write_op.cc:454] Released schema lock
0924 12:48:04.809071 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0924 12:48:04.809172 (+ 101us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:48:04.809176 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0924 12:48:04.632127 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:48:04.632249 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:48:04.632254 (+ 5us) write_op.cc:435] Acquired schema lock
0924 12:48:04.632256 (+ 2us) tablet.cc:660] Decoding operations
0924 12:48:04.632339 (+ 83us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:48:04.632351 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:48:04.632354 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:48:04.632403 (+ 49us) tablet.cc:699] Row locks acquired
0924 12:48:04.632404 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0924 12:48:04.807615 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:48:04.807735 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:48:04.807740 (+ 5us) write_op.cc:435] Acquired schema lock
0924 12:48:04.807742 (+ 2us) tablet.cc:660] Decoding operations
0924 12:48:04.807845 (+ 103us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:48:04.807857 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:48:04.807861 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:48:04.807908 (+ 47us) tablet.cc:699] Row locks acquired
0924 12:48:04.807909 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":292,"threads_started":1,"wal-append.queue_time_us":368,"child_traces":[["op",{"apply.queue_time_us":352,"apply.run_cpu_time_us":635,"apply.run_wall_time_us":636,"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":363,"prepare.run_cpu_time_us":406,"prepare.run_wall_time_us":406,"replication_time_us":1921519,"thread_start_us":467,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":351,"prepare.run_cpu_time_us":364,"prepare.run_wall_time_us":365,"thread_start_us":217,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":70,"prepare.run_cpu_time_us":370,"prepare.run_wall_time_us":370}]]}
I20260924 12:48:05.063273 9268 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:48:05.063499 9268 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:48:05.064252 9268 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:48:05.064858 8731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:48:05.064917 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:48:05.065172 9290 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=46, attempt_no=0}) took 1097 ms. Trace:
I20260924 12:48:05.065507 8795 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:48:05.065301 9290 rpcz_store.cc:276] 0924 12:48:03.967464 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:48:03.967572 (+ 108us) service_pool.cc:227] Handling call
0924 12:48:05.065100 (+1097528us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:48:04.318585 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:48:04.318734 (+ 149us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:48:04.318740 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:48:04.318742 (+ 2us) tablet.cc:660] Decoding operations
0924 12:48:04.318843 (+ 101us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:48:04.318853 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0924 12:48:04.318857 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:48:04.318904 (+ 47us) tablet.cc:699] Row locks acquired
0924 12:48:04.318906 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:48:04.318946 (+ 40us) write_op.cc:270] Start()
0924 12:48:04.694088 (+375142us) write_op.cc:276] Timestamp: P: 1790254084318894 usec, L: 0
0924 12:48:04.694092 (+ 4us) op_driver.cc:331] REPLICATION: starting
0924 12:48:04.694341 (+ 249us) log.cc:842] Serialized 354 byte log entry
0924 12:48:05.064429 (+370088us) op_driver.cc:444] REPLICATION: finished
0924 12:48:05.064549 (+ 120us) write_op.cc:301] APPLY: starting
0924 12:48:05.064569 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0924 12:48:05.064643 (+ 74us) tablet.cc:1368] finished BulkCheckPresence
0924 12:48:05.064663 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:48:05.064744 (+ 81us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:48:05.064777 (+ 33us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:48:05.064784 (+ 7us) write_op.cc:312] APPLY: finished
0924 12:48:05.064890 (+ 106us) log.cc:842] Serialized 43 byte log entry
0924 12:48:05.064959 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:48:05.064978 (+ 19us) write_op.cc:454] Released schema lock
0924 12:48:05.064985 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":63,"num_ops":1,"prepare.queue_time_us":350541,"prepare.run_cpu_time_us":769,"prepare.run_wall_time_us":375836,"replication_time_us":370308}]]}
I20260924 12:48:05.374889 9268 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:48:05.462347 9268 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:48:05.462663 9268 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:48:05.463404 9268 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:48:05.463958 8731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:48:05.464032 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:48:05.464816 8798 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:48:05.465130 9268 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:48:06.040218 9268 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:48:06.040454 9268 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:48:06.041190 9268 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:48:06.041733 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 51 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:48:06.041821 8731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 51 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:48:06.042565 8795 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:48:06.043170 9268 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:48:06.466828 9238 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:48:06.466936 8730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.5.77.195:39535 (request call id 3458) took 1655 ms. Trace:
I20260924 12:48:06.467031 9238 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:48:06.468078 9238 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:48:06.467032 8730 rpcz_store.cc:276] 0924 12:48:04.810977 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:48:04.811037 (+ 60us) service_pool.cc:227] Handling call
0924 12:48:04.811282 (+ 245us) raft_consensus.cc:1457] Updating replica for 4 ops
0924 12:48:04.811410 (+ 128us) raft_consensus.cc:1520] Early marking committed up to index 45
0924 12:48:04.811807 (+ 397us) raft_consensus.cc:1525] Triggering prepare for 4 ops
0924 12:48:04.811964 (+ 157us) write_op.cc:270] Start()
0924 12:48:05.381119 (+569155us) write_op.cc:276] Timestamp: P: 1790254083030269 usec, L: 0
0924 12:48:05.381691 (+ 572us) write_op.cc:270] Start()
0924 12:48:05.524855 (+143164us) write_op.cc:276] Timestamp: P: 1790254083712958 usec, L: 0
0924 12:48:05.525154 (+ 299us) write_op.cc:270] Start()
0924 12:48:05.927363 (+402209us) write_op.cc:276] Timestamp: P: 1790254084318894 usec, L: 0
0924 12:48:05.927752 (+ 389us) write_op.cc:270] Start()
0924 12:48:06.460908 (+533156us) write_op.cc:276] Timestamp: P: 1790254084694787 usec, L: 0
0924 12:48:06.461902 (+ 994us) log.cc:842] Serialized 1416 byte log entry
0924 12:48:06.462292 (+ 390us) raft_consensus.cc:1631] Marking committed up to 47
0924 12:48:06.466706 (+ 4414us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 12:48:06.466743 (+ 37us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 12:48:06.466813 (+ 70us) raft_consensus.cc:1671] finished
0924 12:48:06.466817 (+ 4us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 12:48:06.466879 (+ 62us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:48:05.381585 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:48:05.381713 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:48:05.381719 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:48:05.381721 (+ 2us) tablet.cc:660] Decoding operations
0924 12:48:05.381816 (+ 95us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:48:05.381833 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0924 12:48:05.381838 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:48:05.381892 (+ 54us) tablet.cc:699] Row locks acquired
0924 12:48:05.381894 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:48:06.462335 (+1080441us) op_driver.cc:444] REPLICATION: finished
0924 12:48:06.462890 (+ 555us) write_op.cc:301] APPLY: starting
0924 12:48:06.462908 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0924 12:48:06.462946 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
0924 12:48:06.462947 (+ 1us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:48:06.463046 (+ 99us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:48:06.463073 (+ 27us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:48:06.463134 (+ 61us) write_op.cc:312] APPLY: finished
0924 12:48:06.463231 (+ 97us) log.cc:842] Serialized 43 byte log entry
0924 12:48:06.463291 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:48:06.463313 (+ 22us) write_op.cc:454] Released schema lock
0924 12:48:06.463319 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0924 12:48:06.463412 (+ 93us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:48:06.463415 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0924 12:48:05.525105 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:48:05.525209 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:48:05.525213 (+ 4us) write_op.cc:435] Acquired schema lock
0924 12:48:05.525215 (+ 2us) tablet.cc:660] Decoding operations
0924 12:48:05.525316 (+ 101us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:48:05.525327 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0924 12:48:05.525330 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:48:05.525383 (+ 53us) tablet.cc:699] Row locks acquired
0924 12:48:05.525385 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:48:06.462803 (+937418us) op_driver.cc:444] REPLICATION: finished
0924 12:48:06.463497 (+ 694us) write_op.cc:301] APPLY: starting
0924 12:48:06.463506 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
0924 12:48:06.463542 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0924 12:48:06.463543 (+ 1us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:48:06.463586 (+ 43us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:48:06.463607 (+ 21us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:48:06.463626 (+ 19us) write_op.cc:312] APPLY: finished
0924 12:48:06.463729 (+ 103us) log.cc:842] Serialized 43 byte log entry
0924 12:48:06.463789 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:48:06.463803 (+ 14us) write_op.cc:454] Released schema lock
0924 12:48:06.463810 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0924 12:48:06.463878 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:48:06.463880 (+ 2us) write_op.cc:454] Released schema lock
Related trace 'op':
0924 12:48:05.927642 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:48:05.927772 (+ 130us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:48:05.927779 (+ 7us) write_op.cc:435] Acquired schema lock
0924 12:48:05.927783 (+ 4us) tablet.cc:660] Decoding operations
0924 12:48:05.927866 (+ 83us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:48:05.927879 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0924 12:48:05.927883 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:48:05.927932 (+ 49us) tablet.cc:699] Row locks acquired
0924 12:48:05.927933 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0924 12:48:06.461518 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:48:06.461709 (+ 191us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:48:06.461715 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:48:06.461717 (+ 2us) tablet.cc:660] Decoding operations
0924 12:48:06.461933 (+ 216us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:48:06.461947 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0924 12:48:06.461951 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:48:06.462004 (+ 53us) tablet.cc:699] Row locks acquired
0924 12:48:06.462006 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":286,"threads_started":1,"wal-append.queue_time_us":389,"child_traces":[["op",{"apply.queue_time_us":488,"apply.run_cpu_time_us":585,"apply.run_wall_time_us":586,"num_ops":1,"partition_lock_wait_us":8,"prepare.queue_time_us":336,"prepare.run_cpu_time_us":414,"prepare.run_wall_time_us":413,"replication_time_us":1650356,"thread_start_us":598,"threads_started":2}],["op",{"apply.queue_time_us":664,"apply.run_cpu_time_us":419,"apply.run_wall_time_us":419,"mutex_wait_us":45,"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":52,"prepare.run_cpu_time_us":368,"prepare.run_wall_time_us":367,"replication_time_us":1081098,"thread_start_us":3769,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":105,"prepare.run_cpu_time_us":363,"prepare.run_wall_time_us":363}],["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":423,"prepare.run_cpu_time_us":580,"prepare.run_wall_time_us":579,"thread_start_us":257,"threads_started":1}]]}
I20260924 12:48:06.468516 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 49 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:48:06.468768 8862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 49 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9" is_pre_election: true
I20260924 12:48:06.469296 8666 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0764fc96f6b048d6af98e1e628eef36b; no voters: 34e93f57f57f41d49de37a374ea07bb9, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:48:06.469543 9238 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:48:06.616258 9268 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:48:06.616482 9268 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:48:06.617357 9268 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:48:06.617843 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:48:06.617885 8731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:48:06.618637 8798 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:48:06.618892 9268 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:48:06.618711 9306 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50170 (ReqId={client: c47da85944b049b7aa250e667ddb5e6e, seq_no=50, attempt_no=0}) took 1152 ms. Trace:
I20260924 12:48:06.618991 9306 rpcz_store.cc:276] 0924 12:48:05.465731 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:48:05.465788 (+ 57us) service_pool.cc:227] Handling call
0924 12:48:06.618654 (+1152866us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:48:05.791407 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:48:05.791534 (+ 127us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:48:05.791540 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:48:05.791542 (+ 2us) tablet.cc:660] Decoding operations
0924 12:48:05.791639 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:48:05.791651 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:48:05.791656 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:48:05.791709 (+ 53us) tablet.cc:699] Row locks acquired
0924 12:48:05.791711 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:48:05.791751 (+ 40us) write_op.cc:270] Start()
0924 12:48:06.232905 (+441154us) write_op.cc:276] Timestamp: P: 1790254085791706 usec, L: 0
0924 12:48:06.232911 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:48:06.233159 (+ 248us) log.cc:842] Serialized 354 byte log entry
0924 12:48:06.617538 (+384379us) op_driver.cc:444] REPLICATION: finished
0924 12:48:06.617929 (+ 391us) write_op.cc:301] APPLY: starting
0924 12:48:06.617947 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0924 12:48:06.617990 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0924 12:48:06.618090 (+ 100us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:48:06.618191 (+ 101us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:48:06.618223 (+ 32us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:48:06.618227 (+ 4us) write_op.cc:312] APPLY: finished
0924 12:48:06.618354 (+ 127us) log.cc:842] Serialized 43 byte log entry
0924 12:48:06.618428 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:48:06.618451 (+ 23us) write_op.cc:454] Released schema lock
0924 12:48:06.618482 (+ 31us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":332,"num_ops":1,"prepare.queue_time_us":325243,"prepare.run_cpu_time_us":758,"prepare.run_wall_time_us":441829,"replication_time_us":384599,"thread_start_us":218,"threads_started":1}]]}
I20260924 12:48:06.722492 8842 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260924 12:48:06.724865 8972 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260924 12:48:08.532255 9238 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:48:08.532315 8730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.5.77.195:39535 (request call id 3459) took 2062 ms. Trace:
I20260924 12:48:08.532473 9238 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:48:08.532449 8730 rpcz_store.cc:276] 0924 12:48:06.469717 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:48:06.469765 (+ 48us) service_pool.cc:227] Handling call
0924 12:48:06.469952 (+ 187us) raft_consensus.cc:1457] Updating replica for 3 ops
0924 12:48:06.470053 (+ 101us) raft_consensus.cc:1520] Early marking committed up to index 49
0924 12:48:06.470175 (+ 122us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0924 12:48:06.470284 (+ 109us) write_op.cc:270] Start()
0924 12:48:07.418507 (+948223us) write_op.cc:276] Timestamp: P: 1790254085068060 usec, L: 0
0924 12:48:07.419094 (+ 587us) write_op.cc:270] Start()
0924 12:48:08.069302 (+650208us) write_op.cc:276] Timestamp: P: 1790254085392910 usec, L: 0
0924 12:48:08.070026 (+ 724us) write_op.cc:270] Start()
0924 12:48:08.530204 (+460178us) write_op.cc:276] Timestamp: P: 1790254085791706 usec, L: 0
0924 12:48:08.530623 (+ 419us) log.cc:842] Serialized 1062 byte log entry
0924 12:48:08.530939 (+ 316us) raft_consensus.cc:1631] Marking committed up to 51
0924 12:48:08.532124 (+ 1185us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 12:48:08.532158 (+ 34us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 12:48:08.532170 (+ 12us) raft_consensus.cc:1671] finished
0924 12:48:08.532172 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 12:48:08.532251 (+ 79us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:48:07.418993 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:48:07.419114 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:48:07.419120 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:48:07.419122 (+ 2us) tablet.cc:660] Decoding operations
0924 12:48:07.419216 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:48:07.419230 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0924 12:48:07.419234 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:48:07.419280 (+ 46us) tablet.cc:699] Row locks acquired
0924 12:48:07.419282 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:48:08.530973 (+1111691us) op_driver.cc:444] REPLICATION: finished
0924 12:48:08.531485 (+ 512us) write_op.cc:301] APPLY: starting
0924 12:48:08.531502 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0924 12:48:08.531542 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0924 12:48:08.531545 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:48:08.531644 (+ 99us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:48:08.531678 (+ 34us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:48:08.531712 (+ 34us) write_op.cc:312] APPLY: finished
0924 12:48:08.531836 (+ 124us) log.cc:842] Serialized 43 byte log entry
0924 12:48:08.531903 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:48:08.532152 (+ 249us) write_op.cc:454] Released schema lock
0924 12:48:08.532161 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0924 12:48:08.532302 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:48:08.532306 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0924 12:48:08.069925 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:48:08.070054 (+ 129us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:48:08.070060 (+ 6us) write_op.cc:435] Acquired schema lock
0924 12:48:08.070063 (+ 3us) tablet.cc:660] Decoding operations
0924 12:48:08.070152 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:48:08.070163 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0924 12:48:08.070166 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:48:08.070212 (+ 46us) tablet.cc:699] Row locks acquired
0924 12:48:08.070214 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:48:08.531424 (+461210us) op_driver.cc:444] REPLICATION: finished
0924 12:48:08.532209 (+ 785us) write_op.cc:301] APPLY: starting
0924 12:48:08.532228 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0924 12:48:08.532279 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
0924 12:48:08.532281 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:48:08.532376 (+ 95us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:48:08.532404 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:48:08.532457 (+ 53us) write_op.cc:312] APPLY: finished
0924 12:48:08.532543 (+ 86us) log.cc:842] Serialized 43 byte log entry
0924 12:48:08.532621 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:48:08.532636 (+ 15us) write_op.cc:454] Released schema lock
0924 12:48:08.532642 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0924 12:48:08.532736 (+ 94us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:48:08.532740 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0924 12:48:08.530393 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b077e0d6af50421c8292518a044c561e
0924 12:48:08.530539 (+ 146us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:48:08.530546 (+ 7us) write_op.cc:435] Acquired schema lock
0924 12:48:08.530548 (+ 2us) tablet.cc:660] Decoding operations
0924 12:48:08.530695 (+ 147us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:48:08.530710 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0924 12:48:08.530715 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:48:08.531935 (+ 1220us) tablet.cc:699] Row locks acquired
0924 12:48:08.531939 (+ 4us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":227,"threads_started":1,"wal-append.queue_time_us":323,"child_traces":[["op",{"apply.queue_time_us":449,"apply.run_cpu_time_us":677,"apply.run_wall_time_us":900,"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":350,"prepare.run_cpu_time_us":396,"prepare.run_wall_time_us":395,"replication_time_us":2060675,"thread_start_us":595,"threads_started":2}],["op",{"apply.queue_time_us":700,"apply.run_cpu_time_us":628,"apply.run_wall_time_us":627,"mutex_wait_us":3,"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":487,"prepare.run_cpu_time_us":382,"prepare.run_wall_time_us":381,"replication_time_us":1112314,"thread_start_us":1012,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":599,"prepare.run_wall_time_us":1626,"row_lock_wait_count":1,"row_lock_wait_us":1069}]]}
I20260924 12:48:08.533589 9238 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:48:08.534184 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:48:08.534340 8861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "0764fc96f6b048d6af98e1e628eef36b" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "34e93f57f57f41d49de37a374ea07bb9" is_pre_election: true
I20260924 12:48:08.533610 8691 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:56136 (request call id 1) took 1813 ms. Trace:
I20260924 12:48:08.534622 8691 rpcz_store.cc:276] 0924 12:48:06.719609 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:48:06.719755 (+ 146us) service_pool.cc:227] Handling call
0924 12:48:08.533207 (+1813452us) spinlock_profiling.cc:243] Waited 1.81 s on lock 0x615000082b24. stack: 00007f53d05ab66e 00007f53d05ab430 00007f53cf6325ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007f53dbab7630 00007f53df68fc7f 00007f53da1fb016 00007f53da1fad45 00007f53d70e79be 00007f53d70e679e 00007f53d70ecb25 00007f53d70ed54f 00007f53d70ed378 00000000009edb7c
0924 12:48:08.533497 (+ 290us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4168406400}
I20260924 12:48:08.534982 8666 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0764fc96f6b048d6af98e1e628eef36b; no voters: 34e93f57f57f41d49de37a374ea07bb9, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:48:08.535234 9238 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:48:08.537689 8711 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
e0984e14de3041ddb9233bd3b9f3c1f2 | 127.5.77.254:32945 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.77.212:41043 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0764fc96f6b048d6af98e1e628eef36b | 127.5.77.193:41413 | HEALTHY | <none> | 0 | 0
34e93f57f57f41d49de37a374ea07bb9 | 127.5.77.194:45615 | HEALTHY | <none> | 0 | 0
4b04ce5d5bd546d880ced66f9b15decd | 127.5.77.195:38151 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
inject_latency_ms_before_starting_op | 1000 | hidden,advanced | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.5.77.193 | experimental | 127.5.77.193:41413
local_ip_for_outbound_sockets | 127.5.77.194 | experimental | 127.5.77.194:45615
local_ip_for_outbound_sockets | 127.5.77.195 | experimental | 127.5.77.195:38151
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.5.77.193:41413
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.77.194:45615
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.77.195:38151
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.77.212:41043 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260924 12:48:08.575354 5431 log_verifier.cc:126] Checking tablet b077e0d6af50421c8292518a044c561e
I20260924 12:48:08.636891 5431 log_verifier.cc:177] Verified matching terms for 52 ops in tablet b077e0d6af50421c8292518a044c561e
I20260924 12:48:08.674989 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 8649
I20260924 12:48:08.684828 8771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:08.761860 9352 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:48:08.763044 9352 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:48:08.764667 9352 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:48:08.764931 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
W20260924 12:48:08.770889 8798 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.5.77.193:41413 timed out after 0.006s (SENT) [suppressed 1 similar messages]
W20260924 12:48:08.785706 8798 leader_election.cc:334] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Timed out: RequestConsensusVote RPC to 127.5.77.193:41413 timed out after 0.006s (SENT)
I20260924 12:48:08.785851 8798 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:48:08.786405 9352 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:48:08.842271 9352 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:48:08.842448 9352 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:48:08.843505 9352 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:48:08.844800 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
W20260924 12:48:08.850800 8731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.5.77.194:36097 (request call id 43) took 8 ms (client timeout 6 ms). Trace:
W20260924 12:48:08.856221 8730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.5.77.194:36097 (request call id 42) took 9 ms (client timeout 6 ms). Trace:
W20260924 12:48:08.856375 8730 rpcz_store.cc:269] 0924 12:48:08.841697 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:48:08.850593 (+ 8896us) service_pool.cc:214] Skipping call since client already timed out
0924 12:48:08.850746 (+ 153us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260924 12:48:08.856724 8729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
W20260924 12:48:08.856684 8731 rpcz_store.cc:269] 0924 12:48:08.841832 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:48:08.850585 (+ 8753us) service_pool.cc:214] Skipping call since client already timed out
0924 12:48:08.850747 (+ 162us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260924 12:48:08.857218 8728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "0764fc96f6b048d6af98e1e628eef36b" is_pre_election: true
I20260924 12:48:08.857831 9238 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:48:08.857981 8798 leader_election.cc:302] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e93f57f57f41d49de37a374ea07bb9; no voters: 0764fc96f6b048d6af98e1e628eef36b, 4b04ce5d5bd546d880ced66f9b15decd
I20260924 12:48:08.857968 9238 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:48:08.858592 9238 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 0764fc96f6b048d6af98e1e628eef36b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615)
I20260924 12:48:08.858794 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 8649
I20260924 12:48:08.861184 9352 raft_consensus.cc:2726] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 12:48:08.895484 8928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.77.193:41413 (error 108) [suppressed 1 similar messages]
I20260924 12:48:08.895826 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 8780
I20260924 12:48:08.901399 9352 raft_consensus.cc:482] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b04ce5d5bd546d880ced66f9b15decd)
I20260924 12:48:08.901610 9352 raft_consensus.cc:504] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b04ce5d5bd546d880ced66f9b15decd" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38151 } } peers { permanent_uuid: "0764fc96f6b048d6af98e1e628eef36b" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41413 } } peers { permanent_uuid: "34e93f57f57f41d49de37a374ea07bb9" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 45615 } }
I20260924 12:48:08.902357 9352 leader_election.cc:288] T b077e0d6af50421c8292518a044c561e P 34e93f57f57f41d49de37a374ea07bb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b04ce5d5bd546d880ced66f9b15decd (127.5.77.195:38151), 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413)
I20260924 12:48:08.905864 8902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:08.914186 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
W20260924 12:48:08.914280 8928 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Couldn't send request to peer 0764fc96f6b048d6af98e1e628eef36b. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41413: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 12:48:08.932374 8928 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Couldn't send request to peer 0764fc96f6b048d6af98e1e628eef36b. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41413: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260924 12:48:08.939284 8928 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Couldn't send request to peer 0764fc96f6b048d6af98e1e628eef36b. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41413: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260924 12:48:08.946069 8928 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Couldn't send request to peer 0764fc96f6b048d6af98e1e628eef36b. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41413: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260924 12:48:08.952906 8928 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Couldn't send request to peer 0764fc96f6b048d6af98e1e628eef36b. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41413: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260924 12:48:08.959605 8928 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Couldn't send request to peer 0764fc96f6b048d6af98e1e628eef36b. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41413: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260924 12:48:08.966341 8928 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Couldn't send request to peer 0764fc96f6b048d6af98e1e628eef36b. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41413: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260924 12:48:08.973059 8928 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Couldn't send request to peer 0764fc96f6b048d6af98e1e628eef36b. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41413: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260924 12:48:08.980026 8928 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Couldn't send request to peer 0764fc96f6b048d6af98e1e628eef36b. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41413: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260924 12:48:08.987102 8928 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Couldn't send request to peer 0764fc96f6b048d6af98e1e628eef36b. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41413: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260924 12:48:08.993835 8928 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Couldn't send request to peer 0764fc96f6b048d6af98e1e628eef36b. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41413: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260924 12:48:09.000774 8928 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Couldn't send request to peer 0764fc96f6b048d6af98e1e628eef36b. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41413: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260924 12:48:09.008980 8928 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Couldn't send request to peer 0764fc96f6b048d6af98e1e628eef36b. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41413: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260924 12:48:09.016055 8928 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Couldn't send request to peer 0764fc96f6b048d6af98e1e628eef36b. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41413: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260924 12:48:09.023811 8928 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Couldn't send request to peer 0764fc96f6b048d6af98e1e628eef36b. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41413: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20260924 12:48:09.030515 8928 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Couldn't send request to peer 0764fc96f6b048d6af98e1e628eef36b. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41413: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20260924 12:48:09.037458 8928 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Couldn't send request to peer 0764fc96f6b048d6af98e1e628eef36b. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41413: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20260924 12:48:09.045393 8928 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Couldn't send request to peer 0764fc96f6b048d6af98e1e628eef36b. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41413: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20260924 12:48:09.051997 8928 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Couldn't send request to peer 0764fc96f6b048d6af98e1e628eef36b. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41413: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20260924 12:48:09.058661 8928 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Couldn't send request to peer 0764fc96f6b048d6af98e1e628eef36b. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41413: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20260924 12:48:09.065135 8928 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Couldn't send request to peer 0764fc96f6b048d6af98e1e628eef36b. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41413: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20260924 12:48:09.071905 8928 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Couldn't send request to peer 0764fc96f6b048d6af98e1e628eef36b. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41413: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20260924 12:48:09.078491 8928 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Couldn't send request to peer 0764fc96f6b048d6af98e1e628eef36b. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41413: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20260924 12:48:09.084975 8928 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Couldn't send request to peer 0764fc96f6b048d6af98e1e628eef36b. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41413: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20260924 12:48:09.093463 8928 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Couldn't send request to peer 0764fc96f6b048d6af98e1e628eef36b. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41413: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20260924 12:48:09.099591 8928 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Couldn't send request to peer 0764fc96f6b048d6af98e1e628eef36b. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41413: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20260924 12:48:09.106540 8928 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Couldn't send request to peer 0764fc96f6b048d6af98e1e628eef36b. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41413: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20260924 12:48:09.114850 8928 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Couldn't send request to peer 0764fc96f6b048d6af98e1e628eef36b. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41413: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20260924 12:48:09.115417 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b077e0d6af50421c8292518a044c561e" candidate_uuid: "34e93f57f57f41d49de37a374ea07bb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "4b04ce5d5bd546d880ced66f9b15decd" is_pre_election: true
I20260924 12:48:09.118769 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 8780
W20260924 12:48:09.125576 8928 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Couldn't send request to peer 0764fc96f6b048d6af98e1e628eef36b. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41413: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20260924 12:48:09.131990 8928 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Couldn't send request to peer 0764fc96f6b048d6af98e1e628eef36b. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41413: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20260924 12:48:09.139292 8928 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Couldn't send request to peer 0764fc96f6b048d6af98e1e628eef36b. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41413: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20260924 12:48:09.145017 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 8910
W20260924 12:48:09.146279 8927 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 34e93f57f57f41d49de37a374ea07bb9 (127.5.77.194:45615): Couldn't send request to peer 34e93f57f57f41d49de37a374ea07bb9. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.194:45615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 12:48:09.147182 8928 consensus_peers.cc:487] T b077e0d6af50421c8292518a044c561e P 4b04ce5d5bd546d880ced66f9b15decd -> Peer 0764fc96f6b048d6af98e1e628eef36b (127.5.77.193:41413): Couldn't send request to peer 0764fc96f6b048d6af98e1e628eef36b. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:41413: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20260924 12:48:09.154183 9032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:09.289847 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 8910
I20260924 12:48:09.308106 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 8559
I20260924 12:48:09.309334 8620 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:09.415724 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 8559
2026-09-24T12:48:09Z chronyd exiting
[ OK ] RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys (36041 ms)
[ RUN ] RaftConsensusElectionITest.AutomaticLeaderElection
I20260924 12:48:09.453842 5431 test_util.cc:268] Using random seed: 51331307
I20260924 12:48:09.455497 5431 ts_itest-base.cc:112] Starting cluster with:
I20260924 12:48:09.455586 5431 ts_itest-base.cc:113] --------------
I20260924 12:48:09.455637 5431 ts_itest-base.cc:114] 5 tablet servers
I20260924 12:48:09.455703 5431 ts_itest-base.cc:115] 5 replicas per TS
I20260924 12:48:09.455749 5431 ts_itest-base.cc:116] --------------
2026-09-24T12:48:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:48:09Z Disabled control of system clock
I20260924 12:48:09.483376 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.77.254:34973
--webserver_interface=127.5.77.254
--webserver_port=0
--builtin_ntp_servers=127.5.77.212:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.77.254:34973 with env {}
W20260924 12:48:09.797469 9370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:09.797793 9370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:09.797910 9370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:09.805225 9370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:48:09.805342 9370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:09.805402 9370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:48:09.805477 9370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:48:09.814834 9370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.77.254:34973
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.77.254:34973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.77.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:09.816166 9370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:09.817889 9370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:09.829077 9375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:09.829118 9378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:09.829140 9376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:09.829887 9370 server_base.cc:1034] running on GCE node
I20260924 12:48:09.830642 9370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:09.832165 9370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:09.833376 9370 hybrid_clock.cc:648] HybridClock initialized: now 1790254089833316 us; error 36 us; skew 500 ppm
I20260924 12:48:09.833667 9370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:09.835918 9370 webserver.cc:460] Webserver started at http://127.5.77.254:35091/ using document root <none> and password file <none>
I20260924 12:48:09.836551 9370 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:09.836647 9370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:09.836894 9370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:09.839550 9370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "e46dcba8fbd2492b93dedc7222ac8f54"
format_stamp: "Formatted at 2026-09-24 12:48:09 on dist-test-slave-b8q3"
I20260924 12:48:09.840234 9370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "e46dcba8fbd2492b93dedc7222ac8f54"
format_stamp: "Formatted at 2026-09-24 12:48:09 on dist-test-slave-b8q3"
I20260924 12:48:09.845466 9370 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 12:48:09.848950 9384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:09.850438 9370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 12:48:09.850653 9370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e46dcba8fbd2492b93dedc7222ac8f54"
format_stamp: "Formatted at 2026-09-24 12:48:09 on dist-test-slave-b8q3"
I20260924 12:48:09.850847 9370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:09.870958 9370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:09.872022 9370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:09.872404 9370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:09.895749 9370 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.254:34973
I20260924 12:48:09.895807 9435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.254:34973 every 8 connection(s)
I20260924 12:48:09.897389 9370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260924 12:48:09.901260 9436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:09.901660 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 9370
I20260924 12:48:09.901870 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260924 12:48:09.914798 9436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e46dcba8fbd2492b93dedc7222ac8f54: Bootstrap starting.
I20260924 12:48:09.918186 9436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e46dcba8fbd2492b93dedc7222ac8f54: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:09.919231 9436 log.cc:824] T 00000000000000000000000000000000 P e46dcba8fbd2492b93dedc7222ac8f54: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:09.921692 9436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e46dcba8fbd2492b93dedc7222ac8f54: No bootstrap required, opened a new log
I20260924 12:48:09.928836 9436 raft_consensus.cc:348] T 00000000000000000000000000000000 P e46dcba8fbd2492b93dedc7222ac8f54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e46dcba8fbd2492b93dedc7222ac8f54" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 34973 } }
I20260924 12:48:09.929239 9436 raft_consensus.cc:374] T 00000000000000000000000000000000 P e46dcba8fbd2492b93dedc7222ac8f54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:48:09.929330 9436 raft_consensus.cc:729] T 00000000000000000000000000000000 P e46dcba8fbd2492b93dedc7222ac8f54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e46dcba8fbd2492b93dedc7222ac8f54, State: Initialized, Role: FOLLOWER
I20260924 12:48:09.929867 9436 consensus_queue.cc:260] T 00000000000000000000000000000000 P e46dcba8fbd2492b93dedc7222ac8f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e46dcba8fbd2492b93dedc7222ac8f54" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 34973 } }
I20260924 12:48:09.930085 9436 raft_consensus.cc:388] T 00000000000000000000000000000000 P e46dcba8fbd2492b93dedc7222ac8f54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:48:09.930208 9436 raft_consensus.cc:482] T 00000000000000000000000000000000 P e46dcba8fbd2492b93dedc7222ac8f54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:48:09.930348 9436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e46dcba8fbd2492b93dedc7222ac8f54 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:09.932359 9436 raft_consensus.cc:504] T 00000000000000000000000000000000 P e46dcba8fbd2492b93dedc7222ac8f54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e46dcba8fbd2492b93dedc7222ac8f54" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 34973 } }
I20260924 12:48:09.932835 9436 leader_election.cc:302] T 00000000000000000000000000000000 P e46dcba8fbd2492b93dedc7222ac8f54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e46dcba8fbd2492b93dedc7222ac8f54; no voters:
I20260924 12:48:09.933391 9436 leader_election.cc:288] T 00000000000000000000000000000000 P e46dcba8fbd2492b93dedc7222ac8f54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:48:09.933506 9441 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e46dcba8fbd2492b93dedc7222ac8f54 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:09.934118 9441 raft_consensus.cc:686] T 00000000000000000000000000000000 P e46dcba8fbd2492b93dedc7222ac8f54 [term 1 LEADER]: Becoming Leader. State: Replica: e46dcba8fbd2492b93dedc7222ac8f54, State: Running, Role: LEADER
I20260924 12:48:09.934645 9441 consensus_queue.cc:237] T 00000000000000000000000000000000 P e46dcba8fbd2492b93dedc7222ac8f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e46dcba8fbd2492b93dedc7222ac8f54" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 34973 } }
I20260924 12:48:09.935390 9436 sys_catalog.cc:564] T 00000000000000000000000000000000 P e46dcba8fbd2492b93dedc7222ac8f54 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:48:09.937119 9442 sys_catalog.cc:455] T 00000000000000000000000000000000 P e46dcba8fbd2492b93dedc7222ac8f54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e46dcba8fbd2492b93dedc7222ac8f54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e46dcba8fbd2492b93dedc7222ac8f54" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 34973 } } }
I20260924 12:48:09.937453 9442 sys_catalog.cc:458] T 00000000000000000000000000000000 P e46dcba8fbd2492b93dedc7222ac8f54 [sys.catalog]: This master's current role is: LEADER
I20260924 12:48:09.937602 9443 sys_catalog.cc:455] T 00000000000000000000000000000000 P e46dcba8fbd2492b93dedc7222ac8f54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e46dcba8fbd2492b93dedc7222ac8f54. Latest consensus state: current_term: 1 leader_uuid: "e46dcba8fbd2492b93dedc7222ac8f54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e46dcba8fbd2492b93dedc7222ac8f54" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 34973 } } }
I20260924 12:48:09.937927 9443 sys_catalog.cc:458] T 00000000000000000000000000000000 P e46dcba8fbd2492b93dedc7222ac8f54 [sys.catalog]: This master's current role is: LEADER
I20260924 12:48:09.939307 9446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:48:09.944599 9446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:48:09.953059 9446 catalog_manager.cc:1347] Generated new cluster ID: 3e3c73b493094f0494962f8033db89d8
I20260924 12:48:09.953173 9446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:48:09.970371 9446 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:48:09.972087 9446 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:48:09.982052 9446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e46dcba8fbd2492b93dedc7222ac8f54: Generated new TSK 0
I20260924 12:48:09.983276 9446 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:48:10.000383 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.193:0
--local_ip_for_outbound_sockets=127.5.77.193
--webserver_interface=127.5.77.193
--webserver_port=0
--tserver_master_addrs=127.5.77.254:34973
--builtin_ntp_servers=127.5.77.212:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
W20260924 12:48:10.324113 9460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:10.324410 9460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:10.324508 9460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:10.331727 9460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:10.331918 9460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.193
I20260924 12:48:10.341439 9460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.77.193
--webserver_port=0
--tserver_master_addrs=127.5.77.254:34973
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.193
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:10.342757 9460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:10.344503 9460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:10.356683 9468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:10.356670 9465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:10.357096 9466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:10.357473 9460 server_base.cc:1034] running on GCE node
I20260924 12:48:10.358062 9460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:10.359019 9460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:10.360275 9460 hybrid_clock.cc:648] HybridClock initialized: now 1790254090360189 us; error 49 us; skew 500 ppm
I20260924 12:48:10.360594 9460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:10.366266 9460 webserver.cc:460] Webserver started at http://127.5.77.193:37541/ using document root <none> and password file <none>
I20260924 12:48:10.367008 9460 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:10.367127 9460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:10.367427 9460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:10.370018 9460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa"
format_stamp: "Formatted at 2026-09-24 12:48:10 on dist-test-slave-b8q3"
I20260924 12:48:10.370775 9460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa"
format_stamp: "Formatted at 2026-09-24 12:48:10 on dist-test-slave-b8q3"
I20260924 12:48:10.375727 9460 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 12:48:10.379161 9474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:10.380478 9460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:48:10.380661 9460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa"
format_stamp: "Formatted at 2026-09-24 12:48:10 on dist-test-slave-b8q3"
I20260924 12:48:10.380833 9460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:10.413604 9460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:10.414453 9460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:10.414790 9460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:10.415714 9460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:10.417209 9460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:48:10.417289 9460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:10.417404 9460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:48:10.417454 9460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:10.446530 9460 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.193:34089
I20260924 12:48:10.446580 9586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.193:34089 every 8 connection(s)
I20260924 12:48:10.447795 9460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260924 12:48:10.451563 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 9460
I20260924 12:48:10.451735 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260924 12:48:10.455152 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.194:0
--local_ip_for_outbound_sockets=127.5.77.194
--webserver_interface=127.5.77.194
--webserver_port=0
--tserver_master_addrs=127.5.77.254:34973
--builtin_ntp_servers=127.5.77.212:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20260924 12:48:10.461634 9587 heartbeater.cc:347] Connected to a master server at 127.5.77.254:34973
I20260924 12:48:10.461949 9587 heartbeater.cc:464] Registering TS with master...
I20260924 12:48:10.462821 9587 heartbeater.cc:511] Master 127.5.77.254:34973 requested a full tablet report, sending...
I20260924 12:48:10.465008 9401 ts_manager.cc:194] Registered new tserver with Master: e4fb0f890e4948bbaa5ffcfe269db2aa (127.5.77.193:34089)
I20260924 12:48:10.466648 9401 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.193:42241
W20260924 12:48:10.770036 9591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:10.770336 9591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:10.770437 9591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:10.777768 9591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:10.777966 9591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.194
I20260924 12:48:10.787562 9591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.77.194
--webserver_port=0
--tserver_master_addrs=127.5.77.254:34973
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.194
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:10.788872 9591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:10.790666 9591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:10.803576 9597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:10.803649 9599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:10.803602 9596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:10.804234 9591 server_base.cc:1034] running on GCE node
I20260924 12:48:10.804875 9591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:10.805845 9591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:10.807060 9591 hybrid_clock.cc:648] HybridClock initialized: now 1790254090806978 us; error 55 us; skew 500 ppm
I20260924 12:48:10.807380 9591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:10.809739 9591 webserver.cc:460] Webserver started at http://127.5.77.194:42857/ using document root <none> and password file <none>
I20260924 12:48:10.810426 9591 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:10.810565 9591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:10.810855 9591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:10.813477 9591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "4834837931ea4b4db4517b8089ce7543"
format_stamp: "Formatted at 2026-09-24 12:48:10 on dist-test-slave-b8q3"
I20260924 12:48:10.814217 9591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "4834837931ea4b4db4517b8089ce7543"
format_stamp: "Formatted at 2026-09-24 12:48:10 on dist-test-slave-b8q3"
I20260924 12:48:10.819478 9591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 12:48:10.822840 9605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:10.824234 9591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 12:48:10.824409 9591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "4834837931ea4b4db4517b8089ce7543"
format_stamp: "Formatted at 2026-09-24 12:48:10 on dist-test-slave-b8q3"
I20260924 12:48:10.824594 9591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:10.846575 9591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:10.847427 9591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:10.847729 9591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:10.848757 9591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:10.850284 9591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:48:10.850360 9591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:10.850497 9591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:48:10.850553 9591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260924 12:48:10.884420 9591 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.194:38129
I20260924 12:48:10.884490 9717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.194:38129 every 8 connection(s)
I20260924 12:48:10.885764 9591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260924 12:48:10.895533 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 9591
I20260924 12:48:10.895705 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260924 12:48:10.899189 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.195:0
--local_ip_for_outbound_sockets=127.5.77.195
--webserver_interface=127.5.77.195
--webserver_port=0
--tserver_master_addrs=127.5.77.254:34973
--builtin_ntp_servers=127.5.77.212:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20260924 12:48:10.899644 9718 heartbeater.cc:347] Connected to a master server at 127.5.77.254:34973
I20260924 12:48:10.900041 9718 heartbeater.cc:464] Registering TS with master...
I20260924 12:48:10.901043 9718 heartbeater.cc:511] Master 127.5.77.254:34973 requested a full tablet report, sending...
I20260924 12:48:10.902698 9401 ts_manager.cc:194] Registered new tserver with Master: 4834837931ea4b4db4517b8089ce7543 (127.5.77.194:38129)
I20260924 12:48:10.903817 9401 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.194:57727
W20260924 12:48:11.217743 9721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:11.218037 9721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:11.218116 9721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:11.225307 9721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:11.225479 9721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.195
I20260924 12:48:11.234690 9721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.77.195
--webserver_port=0
--tserver_master_addrs=127.5.77.254:34973
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.195
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:11.236009 9721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:11.237824 9721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:11.250000 9727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:11.250061 9726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:11.250066 9729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:11.250790 9721 server_base.cc:1034] running on GCE node
I20260924 12:48:11.251410 9721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:11.252367 9721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:11.253602 9721 hybrid_clock.cc:648] HybridClock initialized: now 1790254091253485 us; error 76 us; skew 500 ppm
I20260924 12:48:11.253912 9721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:11.256335 9721 webserver.cc:460] Webserver started at http://127.5.77.195:37465/ using document root <none> and password file <none>
I20260924 12:48:11.257038 9721 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:11.257162 9721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:11.257473 9721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:11.260275 9721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "3386972882594ac980b80f630606e76a"
format_stamp: "Formatted at 2026-09-24 12:48:11 on dist-test-slave-b8q3"
I20260924 12:48:11.261054 9721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "3386972882594ac980b80f630606e76a"
format_stamp: "Formatted at 2026-09-24 12:48:11 on dist-test-slave-b8q3"
I20260924 12:48:11.266407 9721 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 12:48:11.269955 9735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:11.271405 9721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:48:11.271560 9721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "3386972882594ac980b80f630606e76a"
format_stamp: "Formatted at 2026-09-24 12:48:11 on dist-test-slave-b8q3"
I20260924 12:48:11.271741 9721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:11.295511 9721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:11.296348 9721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:11.296638 9721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:11.297652 9721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:11.299105 9721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:48:11.299173 9721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:11.299276 9721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:48:11.299324 9721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:11.328363 9721 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.195:35699
I20260924 12:48:11.328409 9847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.195:35699 every 8 connection(s)
I20260924 12:48:11.329597 9721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260924 12:48:11.331462 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 9721
I20260924 12:48:11.331652 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260924 12:48:11.334510 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.196:0
--local_ip_for_outbound_sockets=127.5.77.196
--webserver_interface=127.5.77.196
--webserver_port=0
--tserver_master_addrs=127.5.77.254:34973
--builtin_ntp_servers=127.5.77.212:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20260924 12:48:11.343392 9848 heartbeater.cc:347] Connected to a master server at 127.5.77.254:34973
I20260924 12:48:11.343730 9848 heartbeater.cc:464] Registering TS with master...
I20260924 12:48:11.344592 9848 heartbeater.cc:511] Master 127.5.77.254:34973 requested a full tablet report, sending...
I20260924 12:48:11.346051 9401 ts_manager.cc:194] Registered new tserver with Master: 3386972882594ac980b80f630606e76a (127.5.77.195:35699)
I20260924 12:48:11.346844 9401 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.195:52671
I20260924 12:48:11.470140 9587 heartbeater.cc:503] Master 127.5.77.254:34973 was elected leader, sending a full tablet report...
W20260924 12:48:11.651216 9851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:11.651525 9851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:11.651652 9851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:11.659121 9851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:11.659328 9851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.196
I20260924 12:48:11.668475 9851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.5.77.196
--webserver_port=0
--tserver_master_addrs=127.5.77.254:34973
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.196
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:11.669816 9851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:11.671609 9851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:11.684659 9856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:11.684734 9857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:11.684767 9859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:11.685424 9851 server_base.cc:1034] running on GCE node
I20260924 12:48:11.686034 9851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:11.687031 9851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:11.688249 9851 hybrid_clock.cc:648] HybridClock initialized: now 1790254091688175 us; error 46 us; skew 500 ppm
I20260924 12:48:11.688542 9851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:11.690809 9851 webserver.cc:460] Webserver started at http://127.5.77.196:39019/ using document root <none> and password file <none>
I20260924 12:48:11.691478 9851 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:11.691601 9851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:11.691916 9851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:11.694659 9851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "2f19a213b3d14397b2bd155d1984080b"
format_stamp: "Formatted at 2026-09-24 12:48:11 on dist-test-slave-b8q3"
I20260924 12:48:11.695415 9851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "2f19a213b3d14397b2bd155d1984080b"
format_stamp: "Formatted at 2026-09-24 12:48:11 on dist-test-slave-b8q3"
I20260924 12:48:11.700543 9851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 12:48:11.703963 9865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:11.705369 9851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 12:48:11.705525 9851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "2f19a213b3d14397b2bd155d1984080b"
format_stamp: "Formatted at 2026-09-24 12:48:11 on dist-test-slave-b8q3"
I20260924 12:48:11.705729 9851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:11.719079 9851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:11.719888 9851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:11.720176 9851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:11.721144 9851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:11.722652 9851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:48:11.722733 9851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:11.722854 9851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:48:11.722913 9851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:11.752556 9851 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.196:41513
I20260924 12:48:11.752599 9977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.196:41513 every 8 connection(s)
I20260924 12:48:11.753875 9851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20260924 12:48:11.754379 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 9851
I20260924 12:48:11.754582 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20260924 12:48:11.757740 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.197:0
--local_ip_for_outbound_sockets=127.5.77.197
--webserver_interface=127.5.77.197
--webserver_port=0
--tserver_master_addrs=127.5.77.254:34973
--builtin_ntp_servers=127.5.77.212:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20260924 12:48:11.769536 9978 heartbeater.cc:347] Connected to a master server at 127.5.77.254:34973
I20260924 12:48:11.769887 9978 heartbeater.cc:464] Registering TS with master...
I20260924 12:48:11.770838 9978 heartbeater.cc:511] Master 127.5.77.254:34973 requested a full tablet report, sending...
I20260924 12:48:11.772492 9401 ts_manager.cc:194] Registered new tserver with Master: 2f19a213b3d14397b2bd155d1984080b (127.5.77.196:41513)
I20260924 12:48:11.773617 9401 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.196:49019
I20260924 12:48:11.906488 9718 heartbeater.cc:503] Master 127.5.77.254:34973 was elected leader, sending a full tablet report...
W20260924 12:48:12.081094 9981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:12.081446 9981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:12.081573 9981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:12.089084 9981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:12.089259 9981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.197
I20260924 12:48:12.098353 9981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.5.77.197
--webserver_port=0
--tserver_master_addrs=127.5.77.254:34973
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.197
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:12.099723 9981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:12.101430 9981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:12.114058 9986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:12.114112 9989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:12.114084 9987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:12.114652 9981 server_base.cc:1034] running on GCE node
I20260924 12:48:12.115268 9981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:12.116168 9981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:12.117403 9981 hybrid_clock.cc:648] HybridClock initialized: now 1790254092117338 us; error 32 us; skew 500 ppm
I20260924 12:48:12.117695 9981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:12.119978 9981 webserver.cc:460] Webserver started at http://127.5.77.197:36221/ using document root <none> and password file <none>
I20260924 12:48:12.120674 9981 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:12.120790 9981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:12.121079 9981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:12.123728 9981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6"
format_stamp: "Formatted at 2026-09-24 12:48:12 on dist-test-slave-b8q3"
I20260924 12:48:12.124487 9981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6"
format_stamp: "Formatted at 2026-09-24 12:48:12 on dist-test-slave-b8q3"
I20260924 12:48:12.129869 9981 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 12:48:12.133445 9995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:12.135058 9981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:48:12.135214 9981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6"
format_stamp: "Formatted at 2026-09-24 12:48:12 on dist-test-slave-b8q3"
I20260924 12:48:12.135394 9981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:12.150969 9981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:12.151808 9981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:12.152091 9981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:12.153039 9981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:12.154552 9981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:48:12.154650 9981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:12.154778 9981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:48:12.154829 9981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:12.184185 9981 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.197:39413
I20260924 12:48:12.184265 10107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.197:39413 every 8 connection(s)
I20260924 12:48:12.185563 9981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20260924 12:48:12.189152 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 9981
I20260924 12:48:12.189307 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20260924 12:48:12.199142 10108 heartbeater.cc:347] Connected to a master server at 127.5.77.254:34973
I20260924 12:48:12.199442 10108 heartbeater.cc:464] Registering TS with master...
I20260924 12:48:12.200253 10108 heartbeater.cc:511] Master 127.5.77.254:34973 requested a full tablet report, sending...
I20260924 12:48:12.201689 9401 ts_manager.cc:194] Registered new tserver with Master: 1dbbdaea267a40808a7cfc5ee97bcaf6 (127.5.77.197:39413)
I20260924 12:48:12.202484 9401 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.197:50765
I20260924 12:48:12.204689 5431 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20260924 12:48:12.219933 9401 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49316:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20260924 12:48:12.226662 9401 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 6 tablet servers would be needed, 5 are available. Consider bringing up more tablet servers.
I20260924 12:48:12.274760 9783 tablet_service.cc:1467] Processing CreateTablet for tablet e5419f6c60ca45c894a89c0a2ef92305 (DEFAULT_TABLE table=TestTable [id=471527225cec4cea974785cfc8d6d2ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:48:12.276382 9783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5419f6c60ca45c894a89c0a2ef92305. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:12.276165 10043 tablet_service.cc:1467] Processing CreateTablet for tablet e5419f6c60ca45c894a89c0a2ef92305 (DEFAULT_TABLE table=TestTable [id=471527225cec4cea974785cfc8d6d2ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:48:12.277307 10043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5419f6c60ca45c894a89c0a2ef92305. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:12.277361 9913 tablet_service.cc:1467] Processing CreateTablet for tablet e5419f6c60ca45c894a89c0a2ef92305 (DEFAULT_TABLE table=TestTable [id=471527225cec4cea974785cfc8d6d2ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:48:12.278667 9913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5419f6c60ca45c894a89c0a2ef92305. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:12.288410 10130 tablet_bootstrap.cc:492] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6: Bootstrap starting.
I20260924 12:48:12.288168 9522 tablet_service.cc:1467] Processing CreateTablet for tablet e5419f6c60ca45c894a89c0a2ef92305 (DEFAULT_TABLE table=TestTable [id=471527225cec4cea974785cfc8d6d2ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:48:12.289798 9522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5419f6c60ca45c894a89c0a2ef92305. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:12.271342 9653 tablet_service.cc:1467] Processing CreateTablet for tablet e5419f6c60ca45c894a89c0a2ef92305 (DEFAULT_TABLE table=TestTable [id=471527225cec4cea974785cfc8d6d2ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:48:12.290735 9653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5419f6c60ca45c894a89c0a2ef92305. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:12.293289 10130 tablet_bootstrap.cc:654] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:12.294591 10130 log.cc:824] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:12.305429 10131 tablet_bootstrap.cc:492] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b: Bootstrap starting.
I20260924 12:48:12.307499 10133 tablet_bootstrap.cc:492] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa: Bootstrap starting.
I20260924 12:48:12.308622 10130 tablet_bootstrap.cc:492] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6: No bootstrap required, opened a new log
I20260924 12:48:12.309607 10130 ts_tablet_manager.cc:1397] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.005s
I20260924 12:48:12.311264 10131 tablet_bootstrap.cc:654] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:12.311933 10133 tablet_bootstrap.cc:654] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:12.312644 10131 log.cc:824] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:12.313076 10133 log.cc:824] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:12.319206 10134 tablet_bootstrap.cc:492] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a: Bootstrap starting.
I20260924 12:48:12.321161 10135 tablet_bootstrap.cc:492] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543: Bootstrap starting.
I20260924 12:48:12.322719 10131 tablet_bootstrap.cc:492] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b: No bootstrap required, opened a new log
I20260924 12:48:12.323066 10131 ts_tablet_manager.cc:1397] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.006s
I20260924 12:48:12.323729 10134 tablet_bootstrap.cc:654] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:12.325011 10134 log.cc:824] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:12.325033 10130 raft_consensus.cc:348] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } }
I20260924 12:48:12.325558 10130 raft_consensus.cc:374] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:48:12.325700 10130 raft_consensus.cc:729] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbbdaea267a40808a7cfc5ee97bcaf6, State: Initialized, Role: FOLLOWER
I20260924 12:48:12.325953 10135 tablet_bootstrap.cc:654] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:12.326435 10130 consensus_queue.cc:260] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } }
I20260924 12:48:12.327425 10135 log.cc:824] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:12.328130 10133 tablet_bootstrap.cc:492] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa: No bootstrap required, opened a new log
I20260924 12:48:12.329644 10133 ts_tablet_manager.cc:1397] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.007s
I20260924 12:48:12.337181 10135 tablet_bootstrap.cc:492] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543: No bootstrap required, opened a new log
I20260924 12:48:12.337613 10135 ts_tablet_manager.cc:1397] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20260924 12:48:12.346079 10133 raft_consensus.cc:348] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } }
I20260924 12:48:12.348641 10131 raft_consensus.cc:348] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } }
I20260924 12:48:12.349241 10131 raft_consensus.cc:374] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:48:12.349388 10131 raft_consensus.cc:729] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f19a213b3d14397b2bd155d1984080b, State: Initialized, Role: FOLLOWER
I20260924 12:48:12.349879 9848 heartbeater.cc:503] Master 127.5.77.254:34973 was elected leader, sending a full tablet report...
I20260924 12:48:12.350180 10134 tablet_bootstrap.cc:492] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a: No bootstrap required, opened a new log
I20260924 12:48:12.350490 10134 ts_tablet_manager.cc:1397] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.000s
I20260924 12:48:12.351927 10131 consensus_queue.cc:260] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } }
I20260924 12:48:12.352685 10133 raft_consensus.cc:374] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:48:12.352844 10133 raft_consensus.cc:729] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4fb0f890e4948bbaa5ffcfe269db2aa, State: Initialized, Role: FOLLOWER
I20260924 12:48:12.353694 10133 consensus_queue.cc:260] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } }
I20260924 12:48:12.354766 10108 heartbeater.cc:503] Master 127.5.77.254:34973 was elected leader, sending a full tablet report...
I20260924 12:48:12.351143 10135 raft_consensus.cc:348] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } }
I20260924 12:48:12.355556 10130 ts_tablet_manager.cc:1428] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6: Time spent starting tablet: real 0.044s user 0.017s sys 0.023s
I20260924 12:48:12.355690 10135 raft_consensus.cc:374] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:48:12.355795 10135 raft_consensus.cc:729] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4834837931ea4b4db4517b8089ce7543, State: Initialized, Role: FOLLOWER
I20260924 12:48:12.356465 9978 heartbeater.cc:503] Master 127.5.77.254:34973 was elected leader, sending a full tablet report...
I20260924 12:48:12.356503 10135 consensus_queue.cc:260] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } }
I20260924 12:48:12.357106 10131 ts_tablet_manager.cc:1428] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b: Time spent starting tablet: real 0.034s user 0.011s sys 0.003s
I20260924 12:48:12.361413 10135 ts_tablet_manager.cc:1428] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543: Time spent starting tablet: real 0.024s user 0.014s sys 0.000s
I20260924 12:48:12.363956 10133 ts_tablet_manager.cc:1428] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa: Time spent starting tablet: real 0.034s user 0.006s sys 0.010s
I20260924 12:48:12.365863 10134 raft_consensus.cc:348] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } }
I20260924 12:48:12.366257 10134 raft_consensus.cc:374] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:48:12.366345 10134 raft_consensus.cc:729] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3386972882594ac980b80f630606e76a, State: Initialized, Role: FOLLOWER
I20260924 12:48:12.366954 10134 consensus_queue.cc:260] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } }
I20260924 12:48:12.368458 10134 ts_tablet_manager.cc:1428] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a: Time spent starting tablet: real 0.018s user 0.009s sys 0.003s
W20260924 12:48:12.388198 9719 tablet.cc:2369] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 12:48:12.437098 10109 tablet.cc:2369] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 12:48:12.450757 9588 tablet.cc:2369] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:48:12.487855 10143 raft_consensus.cc:482] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:48:12.488183 10143 raft_consensus.cc:504] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } }
I20260924 12:48:12.490198 10143 leader_election.cc:288] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4fb0f890e4948bbaa5ffcfe269db2aa (127.5.77.193:34089), 1dbbdaea267a40808a7cfc5ee97bcaf6 (127.5.77.197:39413), 3386972882594ac980b80f630606e76a (127.5.77.195:35699), 2f19a213b3d14397b2bd155d1984080b (127.5.77.196:41513)
I20260924 12:48:12.500666 9542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "4834837931ea4b4db4517b8089ce7543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" is_pre_election: true
I20260924 12:48:12.501318 9542 raft_consensus.cc:2454] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4834837931ea4b4db4517b8089ce7543 in term 0.
I20260924 12:48:12.501415 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "4834837931ea4b4db4517b8089ce7543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f19a213b3d14397b2bd155d1984080b" is_pre_election: true
I20260924 12:48:12.501925 9933 raft_consensus.cc:2454] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4834837931ea4b4db4517b8089ce7543 in term 0.
I20260924 12:48:12.502112 10063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "4834837931ea4b4db4517b8089ce7543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" is_pre_election: true
I20260924 12:48:12.502506 9607 leader_election.cc:302] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2f19a213b3d14397b2bd155d1984080b, 4834837931ea4b4db4517b8089ce7543, e4fb0f890e4948bbaa5ffcfe269db2aa; no voters:
I20260924 12:48:12.502647 10063 raft_consensus.cc:2454] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4834837931ea4b4db4517b8089ce7543 in term 0.
I20260924 12:48:12.502938 10143 raft_consensus.cc:2781] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:48:12.503068 10143 raft_consensus.cc:482] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:48:12.502923 9803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "4834837931ea4b4db4517b8089ce7543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3386972882594ac980b80f630606e76a" is_pre_election: true
I20260924 12:48:12.503165 10143 raft_consensus.cc:3037] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:12.503371 9803 raft_consensus.cc:2454] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4834837931ea4b4db4517b8089ce7543 in term 0.
I20260924 12:48:12.505384 10143 raft_consensus.cc:504] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } }
I20260924 12:48:12.506098 10143 leader_election.cc:288] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543 [CANDIDATE]: Term 1 election: Requested vote from peers e4fb0f890e4948bbaa5ffcfe269db2aa (127.5.77.193:34089), 1dbbdaea267a40808a7cfc5ee97bcaf6 (127.5.77.197:39413), 3386972882594ac980b80f630606e76a (127.5.77.195:35699), 2f19a213b3d14397b2bd155d1984080b (127.5.77.196:41513)
W20260924 12:48:12.506119 9979 tablet.cc:2369] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:48:12.506482 9542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "4834837931ea4b4db4517b8089ce7543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa"
I20260924 12:48:12.506656 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "4834837931ea4b4db4517b8089ce7543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f19a213b3d14397b2bd155d1984080b"
I20260924 12:48:12.506698 9542 raft_consensus.cc:3037] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:12.506848 9933 raft_consensus.cc:3037] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:12.507198 10063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "4834837931ea4b4db4517b8089ce7543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6"
I20260924 12:48:12.507239 9803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "4834837931ea4b4db4517b8089ce7543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3386972882594ac980b80f630606e76a"
I20260924 12:48:12.507416 9803 raft_consensus.cc:3037] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:12.507419 10063 raft_consensus.cc:3037] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:12.509709 9542 raft_consensus.cc:2454] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4834837931ea4b4db4517b8089ce7543 in term 1.
I20260924 12:48:12.509743 9933 raft_consensus.cc:2454] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4834837931ea4b4db4517b8089ce7543 in term 1.
I20260924 12:48:12.510244 9608 leader_election.cc:302] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2f19a213b3d14397b2bd155d1984080b, 4834837931ea4b4db4517b8089ce7543, e4fb0f890e4948bbaa5ffcfe269db2aa; no voters:
I20260924 12:48:12.510281 10063 raft_consensus.cc:2454] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4834837931ea4b4db4517b8089ce7543 in term 1.
I20260924 12:48:12.510658 10143 raft_consensus.cc:2781] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:12.511127 10143 raft_consensus.cc:686] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543 [term 1 LEADER]: Becoming Leader. State: Replica: 4834837931ea4b4db4517b8089ce7543, State: Running, Role: LEADER
I20260924 12:48:12.511327 9803 raft_consensus.cc:2454] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4834837931ea4b4db4517b8089ce7543 in term 1.
I20260924 12:48:12.511762 10143 consensus_queue.cc:237] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } }
I20260924 12:48:12.516396 9400 catalog_manager.cc:5495] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4834837931ea4b4db4517b8089ce7543 (127.5.77.194). New cstate: current_term: 1 leader_uuid: "4834837931ea4b4db4517b8089ce7543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } health_report { overall_health: UNKNOWN } } }
I20260924 12:48:12.559746 5431 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20260924 12:48:12.561875 5431 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e4fb0f890e4948bbaa5ffcfe269db2aa to finish bootstrapping
I20260924 12:48:12.570572 5431 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4834837931ea4b4db4517b8089ce7543 to finish bootstrapping
I20260924 12:48:12.578017 5431 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 3386972882594ac980b80f630606e76a to finish bootstrapping
W20260924 12:48:12.581828 9849 tablet.cc:2369] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:48:12.586054 5431 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2f19a213b3d14397b2bd155d1984080b to finish bootstrapping
I20260924 12:48:12.594102 5431 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1dbbdaea267a40808a7cfc5ee97bcaf6 to finish bootstrapping
I20260924 12:48:12.612944 5431 raft_consensus_election-itest.cc:397] Writing data to leader of 5-node config (5 alive)...
I20260924 12:48:12.635778 9803 raft_consensus.cc:1261] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [term 1 FOLLOWER]: Refusing update from remote peer 4834837931ea4b4db4517b8089ce7543: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:48:12.635777 9542 raft_consensus.cc:1261] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [term 1 FOLLOWER]: Refusing update from remote peer 4834837931ea4b4db4517b8089ce7543: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:48:12.635771 9933 raft_consensus.cc:1261] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [term 1 FOLLOWER]: Refusing update from remote peer 4834837931ea4b4db4517b8089ce7543: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:48:12.636103 10063 raft_consensus.cc:1261] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 [term 1 FOLLOWER]: Refusing update from remote peer 4834837931ea4b4db4517b8089ce7543: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:48:12.636973 10149 consensus_queue.cc:1035] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:48:12.637498 10155 consensus_queue.cc:1035] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:48:12.637830 10143 consensus_queue.cc:1035] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:48:12.638062 10156 consensus_queue.cc:1035] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:48:12.664645 10154 mvcc.cc:205] Tried to move back new op lower bound from 7332880763424526336 to 7332880762933198848. Current Snapshot: MvccSnapshot[applied={T|T < 7332880763424526336}]
I20260924 12:48:12.667263 10161 mvcc.cc:205] Tried to move back new op lower bound from 7332880763424526336 to 7332880762933198848. Current Snapshot: MvccSnapshot[applied={T|T < 7332880763424526336}]
I20260924 12:48:12.675107 10157 mvcc.cc:205] Tried to move back new op lower bound from 7332880763424526336 to 7332880762933198848. Current Snapshot: MvccSnapshot[applied={T|T < 7332880763424526336}]
I20260924 12:48:12.720372 5431 raft_consensus_election-itest.cc:408] Killing current leader 4834837931ea4b4db4517b8089ce7543...
I20260924 12:48:12.720534 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 9591
I20260924 12:48:12.721604 9713 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:12.817267 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 9591
I20260924 12:48:12.831080 5431 raft_consensus_election-itest.cc:412] Waiting for new guy to be elected leader.
W20260924 12:48:12.831280 9738 connection.cc:535] server connection from 127.5.77.194:37343 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 12:48:12.833731 5431 test_util.cc:268] Using random seed: 54711189
I20260924 12:48:12.965219 5431 test_util.cc:268] Using random seed: 54842683
I20260924 12:48:13.173987 5431 test_util.cc:268] Using random seed: 55051434
I20260924 12:48:13.481843 5431 test_util.cc:268] Using random seed: 55359318
I20260924 12:48:13.891280 5431 test_util.cc:268] Using random seed: 55768726
I20260924 12:48:14.220458 10173 raft_consensus.cc:482] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4834837931ea4b4db4517b8089ce7543)
I20260924 12:48:14.220681 10173 raft_consensus.cc:504] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } }
I20260924 12:48:14.222306 10174 raft_consensus.cc:482] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4834837931ea4b4db4517b8089ce7543)
I20260924 12:48:14.222481 10175 raft_consensus.cc:482] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4834837931ea4b4db4517b8089ce7543)
I20260924 12:48:14.222550 10174 raft_consensus.cc:504] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } }
I20260924 12:48:14.222674 10175 raft_consensus.cc:504] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } }
I20260924 12:48:14.223399 10173 leader_election.cc:288] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4834837931ea4b4db4517b8089ce7543 (127.5.77.194:38129), 1dbbdaea267a40808a7cfc5ee97bcaf6 (127.5.77.197:39413), 3386972882594ac980b80f630606e76a (127.5.77.195:35699), 2f19a213b3d14397b2bd155d1984080b (127.5.77.196:41513)
W20260924 12:48:14.223464 9475 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.77.194:38129: connect: Connection refused (error 111)
I20260924 12:48:14.248401 10175 leader_election.cc:288] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4834837931ea4b4db4517b8089ce7543 (127.5.77.194:38129), e4fb0f890e4948bbaa5ffcfe269db2aa (127.5.77.193:34089), 3386972882594ac980b80f630606e76a (127.5.77.195:35699), 2f19a213b3d14397b2bd155d1984080b (127.5.77.196:41513)
W20260924 12:48:14.269279 9996 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.77.194:38129: connect: Connection refused (error 111)
I20260924 12:48:14.274631 10174 leader_election.cc:288] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4834837931ea4b4db4517b8089ce7543 (127.5.77.194:38129), e4fb0f890e4948bbaa5ffcfe269db2aa (127.5.77.193:34089), 1dbbdaea267a40808a7cfc5ee97bcaf6 (127.5.77.197:39413), 3386972882594ac980b80f630606e76a (127.5.77.195:35699)
W20260924 12:48:14.277848 9475 leader_election.cc:334] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4834837931ea4b4db4517b8089ce7543 (127.5.77.194:38129): Network error: Client connection negotiation failed: client connection to 127.5.77.194:38129: connect: Connection refused (error 111)
I20260924 12:48:14.278823 9542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" is_pre_election: true
I20260924 12:48:14.279109 9542 raft_consensus.cc:2454] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dbbdaea267a40808a7cfc5ee97bcaf6 in term 1.
I20260924 12:48:14.287232 10063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" is_pre_election: true
I20260924 12:48:14.287528 10063 raft_consensus.cc:2454] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4fb0f890e4948bbaa5ffcfe269db2aa in term 1.
W20260924 12:48:14.287946 9996 leader_election.cc:334] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4834837931ea4b4db4517b8089ce7543 (127.5.77.194:38129): Network error: Client connection negotiation failed: client connection to 127.5.77.194:38129: connect: Connection refused (error 111)
I20260924 12:48:14.289202 9802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "3386972882594ac980b80f630606e76a" is_pre_election: true
I20260924 12:48:14.290182 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "2f19a213b3d14397b2bd155d1984080b" is_pre_election: true
I20260924 12:48:14.290426 9933 raft_consensus.cc:2454] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dbbdaea267a40808a7cfc5ee97bcaf6 in term 1.
I20260924 12:48:14.290930 9997 leader_election.cc:302] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 1dbbdaea267a40808a7cfc5ee97bcaf6, 2f19a213b3d14397b2bd155d1984080b, e4fb0f890e4948bbaa5ffcfe269db2aa; no voters: 4834837931ea4b4db4517b8089ce7543
I20260924 12:48:14.291476 10175 raft_consensus.cc:2781] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 12:48:14.291592 10175 raft_consensus.cc:482] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4834837931ea4b4db4517b8089ce7543)
I20260924 12:48:14.291679 10175 raft_consensus.cc:3037] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:48:14.292069 10063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "2f19a213b3d14397b2bd155d1984080b" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" is_pre_election: true
W20260924 12:48:14.292631 9866 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.77.194:38129: connect: Connection refused (error 111)
I20260924 12:48:14.292800 9802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "3386972882594ac980b80f630606e76a" is_pre_election: true
I20260924 12:48:14.294847 10175 raft_consensus.cc:504] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } }
I20260924 12:48:14.295972 10063 raft_consensus.cc:2379] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2f19a213b3d14397b2bd155d1984080b in current term 2: Already voted for candidate 1dbbdaea267a40808a7cfc5ee97bcaf6 in this term.
I20260924 12:48:14.297542 9542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "2f19a213b3d14397b2bd155d1984080b" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" is_pre_election: true
I20260924 12:48:14.297817 9542 raft_consensus.cc:2454] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f19a213b3d14397b2bd155d1984080b in term 1.
I20260924 12:48:14.298091 9802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "3386972882594ac980b80f630606e76a"
I20260924 12:48:14.298255 10175 leader_election.cc:288] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 [CANDIDATE]: Term 2 election: Requested vote from peers 4834837931ea4b4db4517b8089ce7543 (127.5.77.194:38129), e4fb0f890e4948bbaa5ffcfe269db2aa (127.5.77.193:34089), 3386972882594ac980b80f630606e76a (127.5.77.195:35699), 2f19a213b3d14397b2bd155d1984080b (127.5.77.196:41513)
I20260924 12:48:14.298524 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "2f19a213b3d14397b2bd155d1984080b"
I20260924 12:48:14.298602 9542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa"
I20260924 12:48:14.298734 9933 raft_consensus.cc:3037] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:48:14.298776 9542 raft_consensus.cc:3037] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [term 1 FOLLOWER]: Advancing to term 2
W20260924 12:48:14.300534 9866 leader_election.cc:334] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4834837931ea4b4db4517b8089ce7543 (127.5.77.194:38129): Network error: Client connection negotiation failed: client connection to 127.5.77.194:38129: connect: Connection refused (error 111)
I20260924 12:48:14.301153 9802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "2f19a213b3d14397b2bd155d1984080b" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "3386972882594ac980b80f630606e76a" is_pre_election: true
I20260924 12:48:14.301662 9933 raft_consensus.cc:2454] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dbbdaea267a40808a7cfc5ee97bcaf6 in term 2.
I20260924 12:48:14.301967 9868 leader_election.cc:302] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 2f19a213b3d14397b2bd155d1984080b, e4fb0f890e4948bbaa5ffcfe269db2aa; no voters: 1dbbdaea267a40808a7cfc5ee97bcaf6, 3386972882594ac980b80f630606e76a, 4834837931ea4b4db4517b8089ce7543
I20260924 12:48:14.301718 9542 raft_consensus.cc:2454] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dbbdaea267a40808a7cfc5ee97bcaf6 in term 2.
I20260924 12:48:14.302729 10174 raft_consensus.cc:2726] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:48:14.303112 9997 leader_election.cc:302] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 1dbbdaea267a40808a7cfc5ee97bcaf6, 2f19a213b3d14397b2bd155d1984080b, e4fb0f890e4948bbaa5ffcfe269db2aa; no voters: 3386972882594ac980b80f630606e76a
W20260924 12:48:14.303308 9996 leader_election.cc:334] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4834837931ea4b4db4517b8089ce7543 (127.5.77.194:38129): Network error: Client connection negotiation failed: client connection to 127.5.77.194:38129: connect: Connection refused (error 111)
I20260924 12:48:14.303445 10175 raft_consensus.cc:2781] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 12:48:14.303954 10175 raft_consensus.cc:686] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 [term 2 LEADER]: Becoming Leader. State: Replica: 1dbbdaea267a40808a7cfc5ee97bcaf6, State: Running, Role: LEADER
I20260924 12:48:14.304479 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "2f19a213b3d14397b2bd155d1984080b" is_pre_election: true
I20260924 12:48:14.304514 10175 consensus_queue.cc:237] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } }
I20260924 12:48:14.304738 9933 raft_consensus.cc:2379] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4fb0f890e4948bbaa5ffcfe269db2aa in current term 2: Already voted for candidate 1dbbdaea267a40808a7cfc5ee97bcaf6 in this term.
I20260924 12:48:14.305172 9476 leader_election.cc:302] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 1dbbdaea267a40808a7cfc5ee97bcaf6, e4fb0f890e4948bbaa5ffcfe269db2aa; no voters: 2f19a213b3d14397b2bd155d1984080b, 3386972882594ac980b80f630606e76a, 4834837931ea4b4db4517b8089ce7543
I20260924 12:48:14.305577 10173 raft_consensus.cc:2726] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:48:14.308171 9397 catalog_manager.cc:5495] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 reported cstate change: term changed from 1 to 2, leader changed from 4834837931ea4b4db4517b8089ce7543 (127.5.77.194) to 1dbbdaea267a40808a7cfc5ee97bcaf6 (127.5.77.197). New cstate: current_term: 2 leader_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } health_report { overall_health: UNKNOWN } } }
I20260924 12:48:14.319235 10202 raft_consensus.cc:482] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4834837931ea4b4db4517b8089ce7543)
I20260924 12:48:14.319489 10202 raft_consensus.cc:504] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } }
I20260924 12:48:14.321374 10202 leader_election.cc:288] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4834837931ea4b4db4517b8089ce7543 (127.5.77.194:38129), e4fb0f890e4948bbaa5ffcfe269db2aa (127.5.77.193:34089), 1dbbdaea267a40808a7cfc5ee97bcaf6 (127.5.77.197:39413), 2f19a213b3d14397b2bd155d1984080b (127.5.77.196:41513)
W20260924 12:48:14.322897 9736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.77.194:38129: connect: Connection refused (error 111)
W20260924 12:48:14.328472 9736 leader_election.cc:334] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4834837931ea4b4db4517b8089ce7543 (127.5.77.194:38129): Network error: Client connection negotiation failed: client connection to 127.5.77.194:38129: connect: Connection refused (error 111)
I20260924 12:48:14.331758 9542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "3386972882594ac980b80f630606e76a" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" is_pre_election: true
I20260924 12:48:14.331776 10063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "3386972882594ac980b80f630606e76a" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" is_pre_election: true
I20260924 12:48:14.332104 9542 raft_consensus.cc:2379] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3386972882594ac980b80f630606e76a in current term 2: Already voted for candidate 1dbbdaea267a40808a7cfc5ee97bcaf6 in this term.
I20260924 12:48:14.332729 9738 leader_election.cc:302] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 3386972882594ac980b80f630606e76a; no voters: 1dbbdaea267a40808a7cfc5ee97bcaf6, 4834837931ea4b4db4517b8089ce7543, e4fb0f890e4948bbaa5ffcfe269db2aa
I20260924 12:48:14.333123 10202 raft_consensus.cc:3037] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:48:14.336066 10202 raft_consensus.cc:2726] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:48:14.337117 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "3386972882594ac980b80f630606e76a" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "2f19a213b3d14397b2bd155d1984080b" is_pre_election: true
I20260924 12:48:14.337390 9933 raft_consensus.cc:2379] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3386972882594ac980b80f630606e76a in current term 2: Already voted for candidate 1dbbdaea267a40808a7cfc5ee97bcaf6 in this term.
I20260924 12:48:14.400025 5431 raft_consensus_election-itest.cc:397] Writing data to leader of 5-node config (4 alive)...
W20260924 12:48:14.403652 10111 meta_cache.cc:302] tablet e5419f6c60ca45c894a89c0a2ef92305: replica 4834837931ea4b4db4517b8089ce7543 (127.5.77.194:38129) has failed: Network error: Client connection negotiation failed: client connection to 127.5.77.194:38129: connect: Connection refused (error 111)
W20260924 12:48:14.413646 9502 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43228: Illegal state: replica e4fb0f890e4948bbaa5ffcfe269db2aa is not leader of this config: current role FOLLOWER
I20260924 12:48:14.429476 9802 raft_consensus.cc:1261] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [term 2 FOLLOWER]: Refusing update from remote peer 1dbbdaea267a40808a7cfc5ee97bcaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20260924 12:48:14.429556 9542 raft_consensus.cc:1261] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [term 2 FOLLOWER]: Refusing update from remote peer 1dbbdaea267a40808a7cfc5ee97bcaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20260924 12:48:14.429572 9933 raft_consensus.cc:1261] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [term 2 FOLLOWER]: Refusing update from remote peer 1dbbdaea267a40808a7cfc5ee97bcaf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20260924 12:48:14.430217 10175 consensus_queue.cc:1035] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 12:48:14.430589 10211 consensus_queue.cc:1035] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 12:48:14.430948 10196 consensus_queue.cc:1035] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260924 12:48:14.433720 9996 consensus_peers.cc:487] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 -> Peer 4834837931ea4b4db4517b8089ce7543 (127.5.77.194:38129): Couldn't send request to peer 4834837931ea4b4db4517b8089ce7543. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.194:38129: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 12:48:14.444478 10212 mvcc.cc:205] Tried to move back new op lower bound from 7332880770775584768 to 7332880770276384768. Current Snapshot: MvccSnapshot[applied={T|T < 7332880770775584768}]
I20260924 12:48:14.482864 5431 raft_consensus_election-itest.cc:408] Killing current leader 1dbbdaea267a40808a7cfc5ee97bcaf6...
I20260924 12:48:14.483017 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 9981
I20260924 12:48:14.484133 10103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:14.577636 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 9981
I20260924 12:48:14.591300 5431 raft_consensus_election-itest.cc:412] Waiting for new guy to be elected leader.
I20260924 12:48:14.594164 5431 test_util.cc:268] Using random seed: 56471598
I20260924 12:48:14.702147 5431 test_util.cc:268] Using random seed: 56579618
I20260924 12:48:14.910099 5431 test_util.cc:268] Using random seed: 56787550
I20260924 12:48:15.217816 5431 test_util.cc:268] Using random seed: 57095286
I20260924 12:48:15.625270 5431 test_util.cc:268] Using random seed: 57502718
I20260924 12:48:15.983176 10224 raft_consensus.cc:482] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dbbdaea267a40808a7cfc5ee97bcaf6)
I20260924 12:48:15.983557 10225 raft_consensus.cc:482] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dbbdaea267a40808a7cfc5ee97bcaf6)
I20260924 12:48:15.983435 10224 raft_consensus.cc:504] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } }
I20260924 12:48:15.983721 10225 raft_consensus.cc:504] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } }
I20260924 12:48:15.984401 10224 leader_election.cc:288] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4834837931ea4b4db4517b8089ce7543 (127.5.77.194:38129), 1dbbdaea267a40808a7cfc5ee97bcaf6 (127.5.77.197:39413), 3386972882594ac980b80f630606e76a (127.5.77.195:35699), 2f19a213b3d14397b2bd155d1984080b (127.5.77.196:41513)
I20260924 12:48:15.988400 9542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "3386972882594ac980b80f630606e76a" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" is_pre_election: true
I20260924 12:48:15.988751 9542 raft_consensus.cc:2454] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3386972882594ac980b80f630606e76a in term 2.
I20260924 12:48:15.990713 10225 leader_election.cc:288] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4834837931ea4b4db4517b8089ce7543 (127.5.77.194:38129), e4fb0f890e4948bbaa5ffcfe269db2aa (127.5.77.193:34089), 1dbbdaea267a40808a7cfc5ee97bcaf6 (127.5.77.197:39413), 2f19a213b3d14397b2bd155d1984080b (127.5.77.196:41513)
I20260924 12:48:15.993901 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "2f19a213b3d14397b2bd155d1984080b" is_pre_election: true
I20260924 12:48:15.994236 9933 raft_consensus.cc:2454] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4fb0f890e4948bbaa5ffcfe269db2aa in term 2.
W20260924 12:48:16.001191 9475 leader_election.cc:334] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4834837931ea4b4db4517b8089ce7543 (127.5.77.194:38129): Network error: Client connection negotiation failed: client connection to 127.5.77.194:38129: connect: Connection refused (error 111)
W20260924 12:48:16.001444 9476 leader_election.cc:334] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1dbbdaea267a40808a7cfc5ee97bcaf6 (127.5.77.197:39413): Network error: Client connection negotiation failed: client connection to 127.5.77.197:39413: connect: Connection refused (error 111)
I20260924 12:48:16.001345 9802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "3386972882594ac980b80f630606e76a" is_pre_election: true
I20260924 12:48:16.001688 9802 raft_consensus.cc:2454] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4fb0f890e4948bbaa5ffcfe269db2aa in term 2.
I20260924 12:48:16.001883 10226 raft_consensus.cc:482] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dbbdaea267a40808a7cfc5ee97bcaf6)
I20260924 12:48:16.002058 10226 raft_consensus.cc:504] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } }
I20260924 12:48:16.002215 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "3386972882594ac980b80f630606e76a" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "2f19a213b3d14397b2bd155d1984080b" is_pre_election: true
I20260924 12:48:16.002619 9933 raft_consensus.cc:2454] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3386972882594ac980b80f630606e76a in term 2.
I20260924 12:48:16.002676 9478 leader_election.cc:302] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 5 voters: 3 yes votes; 2 no votes. yes voters: 2f19a213b3d14397b2bd155d1984080b, 3386972882594ac980b80f630606e76a, e4fb0f890e4948bbaa5ffcfe269db2aa; no voters: 1dbbdaea267a40808a7cfc5ee97bcaf6, 4834837931ea4b4db4517b8089ce7543
I20260924 12:48:16.003199 9737 leader_election.cc:302] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2f19a213b3d14397b2bd155d1984080b, 3386972882594ac980b80f630606e76a, e4fb0f890e4948bbaa5ffcfe269db2aa; no voters:
I20260924 12:48:16.003728 10225 raft_consensus.cc:2781] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260924 12:48:16.003742 9737 leader_election.cc:334] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1dbbdaea267a40808a7cfc5ee97bcaf6 (127.5.77.197:39413): Network error: Client connection negotiation failed: client connection to 127.5.77.197:39413: connect: Connection refused (error 111)
I20260924 12:48:16.003896 10225 raft_consensus.cc:482] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1dbbdaea267a40808a7cfc5ee97bcaf6)
I20260924 12:48:16.003999 10225 raft_consensus.cc:3037] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [term 2 FOLLOWER]: Advancing to term 3
W20260924 12:48:16.004361 9736 leader_election.cc:334] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4834837931ea4b4db4517b8089ce7543 (127.5.77.194:38129): Network error: Client connection negotiation failed: client connection to 127.5.77.194:38129: connect: Connection refused (error 111)
I20260924 12:48:16.004762 10224 raft_consensus.cc:2781] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 12:48:16.004906 10224 raft_consensus.cc:482] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1dbbdaea267a40808a7cfc5ee97bcaf6)
I20260924 12:48:16.005005 10224 raft_consensus.cc:3037] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [term 2 FOLLOWER]: Advancing to term 3
I20260924 12:48:16.007153 10225 raft_consensus.cc:504] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } }
I20260924 12:48:16.007964 9542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "2f19a213b3d14397b2bd155d1984080b" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" is_pre_election: true
I20260924 12:48:16.009044 9541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "3386972882594ac980b80f630606e76a" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa"
I20260924 12:48:16.009407 10225 leader_election.cc:288] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [CANDIDATE]: Term 3 election: Requested vote from peers 4834837931ea4b4db4517b8089ce7543 (127.5.77.194:38129), e4fb0f890e4948bbaa5ffcfe269db2aa (127.5.77.193:34089), 1dbbdaea267a40808a7cfc5ee97bcaf6 (127.5.77.197:39413), 2f19a213b3d14397b2bd155d1984080b (127.5.77.196:41513)
I20260924 12:48:16.010011 10226 leader_election.cc:288] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4834837931ea4b4db4517b8089ce7543 (127.5.77.194:38129), e4fb0f890e4948bbaa5ffcfe269db2aa (127.5.77.193:34089), 1dbbdaea267a40808a7cfc5ee97bcaf6 (127.5.77.197:39413), 3386972882594ac980b80f630606e76a (127.5.77.195:35699)
I20260924 12:48:16.010162 9802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "2f19a213b3d14397b2bd155d1984080b" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "3386972882594ac980b80f630606e76a" is_pre_election: true
I20260924 12:48:16.010429 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "3386972882594ac980b80f630606e76a" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "2f19a213b3d14397b2bd155d1984080b"
I20260924 12:48:16.010563 9802 raft_consensus.cc:2379] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2f19a213b3d14397b2bd155d1984080b in current term 3: Already voted for candidate 3386972882594ac980b80f630606e76a in this term.
I20260924 12:48:16.010664 9933 raft_consensus.cc:3037] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [term 2 FOLLOWER]: Advancing to term 3
W20260924 12:48:16.010885 9737 leader_election.cc:334] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1dbbdaea267a40808a7cfc5ee97bcaf6 (127.5.77.197:39413): Network error: Client connection negotiation failed: client connection to 127.5.77.197:39413: connect: Connection refused (error 111)
W20260924 12:48:16.011837 9736 leader_election.cc:334] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4834837931ea4b4db4517b8089ce7543 (127.5.77.194:38129): Network error: Client connection negotiation failed: client connection to 127.5.77.194:38129: connect: Connection refused (error 111)
I20260924 12:48:16.012015 9736 leader_election.cc:302] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 3386972882594ac980b80f630606e76a; no voters: 1dbbdaea267a40808a7cfc5ee97bcaf6, 4834837931ea4b4db4517b8089ce7543, e4fb0f890e4948bbaa5ffcfe269db2aa
I20260924 12:48:16.012349 10225 raft_consensus.cc:2726] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260924 12:48:16.008419 10224 raft_consensus.cc:504] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } }
I20260924 12:48:16.012969 9542 raft_consensus.cc:2379] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2f19a213b3d14397b2bd155d1984080b in current term 3: Already voted for candidate e4fb0f890e4948bbaa5ffcfe269db2aa in this term.
I20260924 12:48:16.014047 9933 raft_consensus.cc:2454] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3386972882594ac980b80f630606e76a in term 3.
I20260924 12:48:16.014369 10224 leader_election.cc:288] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [CANDIDATE]: Term 3 election: Requested vote from peers 4834837931ea4b4db4517b8089ce7543 (127.5.77.194:38129), 1dbbdaea267a40808a7cfc5ee97bcaf6 (127.5.77.197:39413), 3386972882594ac980b80f630606e76a (127.5.77.195:35699), 2f19a213b3d14397b2bd155d1984080b (127.5.77.196:41513)
W20260924 12:48:16.014514 9866 leader_election.cc:334] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4834837931ea4b4db4517b8089ce7543 (127.5.77.194:38129): Network error: Client connection negotiation failed: client connection to 127.5.77.194:38129: connect: Connection refused (error 111)
I20260924 12:48:16.014683 9866 leader_election.cc:302] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 2f19a213b3d14397b2bd155d1984080b; no voters: 3386972882594ac980b80f630606e76a, 4834837931ea4b4db4517b8089ce7543, e4fb0f890e4948bbaa5ffcfe269db2aa
I20260924 12:48:16.015033 9802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "3386972882594ac980b80f630606e76a"
I20260924 12:48:16.015316 9802 raft_consensus.cc:2379] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e4fb0f890e4948bbaa5ffcfe269db2aa in current term 3: Already voted for candidate 3386972882594ac980b80f630606e76a in this term.
W20260924 12:48:16.015293 9867 leader_election.cc:334] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1dbbdaea267a40808a7cfc5ee97bcaf6 (127.5.77.197:39413): Network error: Client connection negotiation failed: client connection to 127.5.77.197:39413: connect: Connection refused (error 111)
I20260924 12:48:16.015753 10226 raft_consensus.cc:2726] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 12:48:16.015698 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "2f19a213b3d14397b2bd155d1984080b"
I20260924 12:48:16.016022 9933 raft_consensus.cc:2379] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e4fb0f890e4948bbaa5ffcfe269db2aa in current term 3: Already voted for candidate 3386972882594ac980b80f630606e76a in this term.
W20260924 12:48:16.016054 9476 leader_election.cc:334] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1dbbdaea267a40808a7cfc5ee97bcaf6 (127.5.77.197:39413): Network error: Client connection negotiation failed: client connection to 127.5.77.197:39413: connect: Connection refused (error 111)
I20260924 12:48:16.016355 9476 leader_election.cc:302] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: e4fb0f890e4948bbaa5ffcfe269db2aa; no voters: 1dbbdaea267a40808a7cfc5ee97bcaf6, 2f19a213b3d14397b2bd155d1984080b, 3386972882594ac980b80f630606e76a
I20260924 12:48:16.016660 10224 raft_consensus.cc:2726] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260924 12:48:16.016670 9475 leader_election.cc:334] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4834837931ea4b4db4517b8089ce7543 (127.5.77.194:38129): Network error: Client connection negotiation failed: client connection to 127.5.77.194:38129: connect: Connection refused (error 111)
I20260924 12:48:16.133023 5431 test_util.cc:268] Using random seed: 58010498
I20260924 12:48:16.741478 5431 test_util.cc:268] Using random seed: 58618932
I20260924 12:48:17.449887 5431 test_util.cc:268] Using random seed: 59327360
I20260924 12:48:17.657877 10239 raft_consensus.cc:482] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:48:17.658047 10239 raft_consensus.cc:504] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } }
I20260924 12:48:17.658946 10239 leader_election.cc:288] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4834837931ea4b4db4517b8089ce7543 (127.5.77.194:38129), e4fb0f890e4948bbaa5ffcfe269db2aa (127.5.77.193:34089), 1dbbdaea267a40808a7cfc5ee97bcaf6 (127.5.77.197:39413), 2f19a213b3d14397b2bd155d1984080b (127.5.77.196:41513)
I20260924 12:48:17.659446 9542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "3386972882594ac980b80f630606e76a" candidate_term: 4 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" is_pre_election: true
I20260924 12:48:17.659694 9542 raft_consensus.cc:2454] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3386972882594ac980b80f630606e76a in term 3.
I20260924 12:48:17.660353 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "3386972882594ac980b80f630606e76a" candidate_term: 4 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "2f19a213b3d14397b2bd155d1984080b" is_pre_election: true
I20260924 12:48:17.660583 9933 raft_consensus.cc:2454] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3386972882594ac980b80f630606e76a in term 3.
I20260924 12:48:17.661168 9737 leader_election.cc:302] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2f19a213b3d14397b2bd155d1984080b, 3386972882594ac980b80f630606e76a, e4fb0f890e4948bbaa5ffcfe269db2aa; no voters:
W20260924 12:48:17.661366 9736 leader_election.cc:334] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4834837931ea4b4db4517b8089ce7543 (127.5.77.194:38129): Network error: Client connection negotiation failed: client connection to 127.5.77.194:38129: connect: Connection refused (error 111)
I20260924 12:48:17.661438 10239 raft_consensus.cc:2781] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260924 12:48:17.661540 9737 leader_election.cc:334] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1dbbdaea267a40808a7cfc5ee97bcaf6 (127.5.77.197:39413): Network error: Client connection negotiation failed: client connection to 127.5.77.197:39413: connect: Connection refused (error 111)
I20260924 12:48:17.661590 10239 raft_consensus.cc:482] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:48:17.661684 10239 raft_consensus.cc:3037] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [term 3 FOLLOWER]: Advancing to term 4
I20260924 12:48:17.663599 10239 raft_consensus.cc:504] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } }
I20260924 12:48:17.664295 10239 leader_election.cc:288] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [CANDIDATE]: Term 4 election: Requested vote from peers 4834837931ea4b4db4517b8089ce7543 (127.5.77.194:38129), e4fb0f890e4948bbaa5ffcfe269db2aa (127.5.77.193:34089), 1dbbdaea267a40808a7cfc5ee97bcaf6 (127.5.77.197:39413), 2f19a213b3d14397b2bd155d1984080b (127.5.77.196:41513)
I20260924 12:48:17.664745 9542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "3386972882594ac980b80f630606e76a" candidate_term: 4 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa"
I20260924 12:48:17.664961 9542 raft_consensus.cc:3037] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [term 3 FOLLOWER]: Advancing to term 4
I20260924 12:48:17.665132 9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5419f6c60ca45c894a89c0a2ef92305" candidate_uuid: "3386972882594ac980b80f630606e76a" candidate_term: 4 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "2f19a213b3d14397b2bd155d1984080b"
I20260924 12:48:17.665366 9933 raft_consensus.cc:3037] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [term 3 FOLLOWER]: Advancing to term 4
W20260924 12:48:17.665813 9737 leader_election.cc:334] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 1dbbdaea267a40808a7cfc5ee97bcaf6 (127.5.77.197:39413): Network error: Client connection negotiation failed: client connection to 127.5.77.197:39413: connect: Connection refused (error 111)
W20260924 12:48:17.666301 9736 leader_election.cc:334] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 4834837931ea4b4db4517b8089ce7543 (127.5.77.194:38129): Network error: Client connection negotiation failed: client connection to 127.5.77.194:38129: connect: Connection refused (error 111)
I20260924 12:48:17.667250 9542 raft_consensus.cc:2454] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3386972882594ac980b80f630606e76a in term 4.
I20260924 12:48:17.667470 9933 raft_consensus.cc:2454] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3386972882594ac980b80f630606e76a in term 4.
I20260924 12:48:17.667881 9737 leader_election.cc:302] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 5 voters: 3 yes votes; 2 no votes. yes voters: 2f19a213b3d14397b2bd155d1984080b, 3386972882594ac980b80f630606e76a, e4fb0f890e4948bbaa5ffcfe269db2aa; no voters: 1dbbdaea267a40808a7cfc5ee97bcaf6, 4834837931ea4b4db4517b8089ce7543
I20260924 12:48:17.668125 10239 raft_consensus.cc:2781] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [term 4 FOLLOWER]: Leader election won for term 4
I20260924 12:48:17.668496 10239 raft_consensus.cc:686] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [term 4 LEADER]: Becoming Leader. State: Replica: 3386972882594ac980b80f630606e76a, State: Running, Role: LEADER
I20260924 12:48:17.668927 10239 consensus_queue.cc:237] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } }
I20260924 12:48:17.672363 9397 catalog_manager.cc:5495] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a reported cstate change: term changed from 2 to 4, leader changed from 1dbbdaea267a40808a7cfc5ee97bcaf6 (127.5.77.197) to 3386972882594ac980b80f630606e76a (127.5.77.195). New cstate: current_term: 4 leader_uuid: "3386972882594ac980b80f630606e76a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } health_report { overall_health: UNKNOWN } } }
I20260924 12:48:18.068571 9933 raft_consensus.cc:1261] T e5419f6c60ca45c894a89c0a2ef92305 P 2f19a213b3d14397b2bd155d1984080b [term 4 FOLLOWER]: Refusing update from remote peer 3386972882594ac980b80f630606e76a: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 4 index: 13. (index mismatch)
I20260924 12:48:18.069264 10239 consensus_queue.cc:1035] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260924 12:48:18.150715 9736 consensus_peers.cc:487] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a -> Peer 4834837931ea4b4db4517b8089ce7543 (127.5.77.194:38129): Couldn't send request to peer 4834837931ea4b4db4517b8089ce7543. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.194:38129: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 12:48:18.154731 9737 consensus_peers.cc:487] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a -> Peer 1dbbdaea267a40808a7cfc5ee97bcaf6 (127.5.77.197:39413): Couldn't send request to peer 1dbbdaea267a40808a7cfc5ee97bcaf6. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.197:39413: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 12:48:18.217514 9542 raft_consensus.cc:1261] T e5419f6c60ca45c894a89c0a2ef92305 P e4fb0f890e4948bbaa5ffcfe269db2aa [term 4 FOLLOWER]: Refusing update from remote peer 3386972882594ac980b80f630606e76a: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 4 index: 13. (index mismatch)
I20260924 12:48:18.218170 10239 consensus_queue.cc:1035] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 12:48:18.258340 5431 raft_consensus_election-itest.cc:397] Writing data to leader of 5-node config (3 alive)...
W20260924 12:48:18.261580 10112 meta_cache.cc:302] tablet e5419f6c60ca45c894a89c0a2ef92305: replica 1dbbdaea267a40808a7cfc5ee97bcaf6 (127.5.77.197:39413) has failed: Network error: Client connection negotiation failed: client connection to 127.5.77.197:39413: connect: Connection refused (error 111)
W20260924 12:48:18.268097 9502 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43228: Illegal state: replica e4fb0f890e4948bbaa5ffcfe269db2aa is not leader of this config: current role FOLLOWER
I20260924 12:48:18.313133 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.197:39413
--local_ip_for_outbound_sockets=127.5.77.197
--tserver_master_addrs=127.5.77.254:34973
--webserver_port=36221
--webserver_interface=127.5.77.197
--builtin_ntp_servers=127.5.77.212:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
W20260924 12:48:18.632604 10266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:18.632882 10266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:18.632969 10266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:18.640920 10266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:18.641151 10266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.197
I20260924 12:48:18.650635 10266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.197:39413
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.5.77.197
--webserver_port=36221
--tserver_master_addrs=127.5.77.254:34973
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.197
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:18.651878 10266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:18.653501 10266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:18.666316 10274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:18.666302 10271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:18.666365 10272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:18.666980 10266 server_base.cc:1034] running on GCE node
I20260924 12:48:18.667610 10266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:18.668463 10266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:18.669674 10266 hybrid_clock.cc:648] HybridClock initialized: now 1790254098669606 us; error 45 us; skew 500 ppm
I20260924 12:48:18.670009 10266 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:18.678748 10266 webserver.cc:460] Webserver started at http://127.5.77.197:36221/ using document root <none> and password file <none>
I20260924 12:48:18.679477 10266 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:18.679605 10266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:18.684887 10266 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 12:48:18.687662 10280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:18.688966 10266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:48:18.689122 10266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6"
format_stamp: "Formatted at 2026-09-24 12:48:12 on dist-test-slave-b8q3"
I20260924 12:48:18.690035 10266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:18.701493 10266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:18.702266 10266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:18.702538 10266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:18.703430 10266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:18.705165 10287 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260924 12:48:18.710520 10266 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 12:48:18.710664 10266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260924 12:48:18.710786 10266 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 12:48:18.713806 10266 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 12:48:18.713889 10266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260924 12:48:18.714191 10287 tablet_bootstrap.cc:492] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6: Bootstrap starting.
I20260924 12:48:18.736428 10287 log.cc:824] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:18.751233 10266 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.197:39413
I20260924 12:48:18.751271 10394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.197:39413 every 8 connection(s)
I20260924 12:48:18.752458 10266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20260924 12:48:18.761932 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 10266
I20260924 12:48:18.762696 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.194:38129
--local_ip_for_outbound_sockets=127.5.77.194
--tserver_master_addrs=127.5.77.254:34973
--webserver_port=42857
--webserver_interface=127.5.77.194
--builtin_ntp_servers=127.5.77.212:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20260924 12:48:18.775550 10287 tablet_bootstrap.cc:492] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 12:48:18.776309 10287 tablet_bootstrap.cc:492] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6: Bootstrap complete.
I20260924 12:48:18.777733 10287 ts_tablet_manager.cc:1397] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6: Time spent bootstrapping tablet: real 0.064s user 0.039s sys 0.012s
I20260924 12:48:18.785706 10395 heartbeater.cc:347] Connected to a master server at 127.5.77.254:34973
I20260924 12:48:18.786023 10395 heartbeater.cc:464] Registering TS with master...
I20260924 12:48:18.785423 10287 raft_consensus.cc:348] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } }
I20260924 12:48:18.786842 10287 raft_consensus.cc:729] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbbdaea267a40808a7cfc5ee97bcaf6, State: Initialized, Role: FOLLOWER
I20260924 12:48:18.786883 10395 heartbeater.cc:511] Master 127.5.77.254:34973 requested a full tablet report, sending...
I20260924 12:48:18.787695 10287 consensus_queue.cc:260] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } }
I20260924 12:48:18.789712 10287 ts_tablet_manager.cc:1428] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260924 12:48:18.791615 9397 ts_manager.cc:194] Re-registered known tserver with Master: 1dbbdaea267a40808a7cfc5ee97bcaf6 (127.5.77.197:39413)
I20260924 12:48:18.793105 9397 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.197:45139
I20260924 12:48:18.796190 10395 heartbeater.cc:503] Master 127.5.77.254:34973 was elected leader, sending a full tablet report...
I20260924 12:48:19.094547 10350 raft_consensus.cc:3037] T e5419f6c60ca45c894a89c0a2ef92305 P 1dbbdaea267a40808a7cfc5ee97bcaf6 [term 2 FOLLOWER]: Advancing to term 4
I20260924 12:48:19.103914 10405 mvcc.cc:205] Tried to move back new op lower bound from 7332880786564071424 to 7332880784056426496. Current Snapshot: MvccSnapshot[applied={T|T < 7332880786564071424}]
W20260924 12:48:19.108517 10399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:19.108832 10399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:19.108963 10399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:19.117502 10399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:19.117717 10399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.194
I20260924 12:48:19.126924 10399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.194:38129
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.77.194
--webserver_port=42857
--tserver_master_addrs=127.5.77.254:34973
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.194
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:19.128185 10399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:19.129863 10399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:19.142521 10412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:19.142555 10415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:19.142576 10413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:19.143386 10399 server_base.cc:1034] running on GCE node
I20260924 12:48:19.144127 10399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:19.145202 10399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:19.146437 10399 hybrid_clock.cc:648] HybridClock initialized: now 1790254099146355 us; error 58 us; skew 500 ppm
I20260924 12:48:19.146775 10399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:19.149202 10399 webserver.cc:460] Webserver started at http://127.5.77.194:42857/ using document root <none> and password file <none>
I20260924 12:48:19.150025 10399 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:19.150169 10399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:19.155543 10399 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 12:48:19.158593 10421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:19.160022 10399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 12:48:19.160210 10399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "4834837931ea4b4db4517b8089ce7543"
format_stamp: "Formatted at 2026-09-24 12:48:10 on dist-test-slave-b8q3"
I20260924 12:48:19.161214 10399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:19.192342 10399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:19.193212 10399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:19.193506 10399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:19.194543 10399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:19.196449 10428 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260924 12:48:19.201996 10399 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 12:48:19.202113 10399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260924 12:48:19.202243 10399 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 12:48:19.205300 10399 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 12:48:19.205372 10399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260924 12:48:19.205730 10428 tablet_bootstrap.cc:492] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543: Bootstrap starting.
I20260924 12:48:19.230049 10428 log.cc:824] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:19.242316 10399 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.194:38129
I20260924 12:48:19.242352 10536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.194:38129 every 8 connection(s)
I20260924 12:48:19.243638 10399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260924 12:48:19.249610 10428 tablet_bootstrap.cc:492] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=50 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 12:48:19.250083 10428 tablet_bootstrap.cc:492] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543: Bootstrap complete.
I20260924 12:48:19.251045 10428 ts_tablet_manager.cc:1397] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543: Time spent bootstrapping tablet: real 0.046s user 0.031s sys 0.012s
I20260924 12:48:19.252022 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 10399
I20260924 12:48:19.259205 10428 raft_consensus.cc:348] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } }
I20260924 12:48:19.260613 10428 raft_consensus.cc:729] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4834837931ea4b4db4517b8089ce7543, State: Initialized, Role: FOLLOWER
I20260924 12:48:19.261257 10537 heartbeater.cc:347] Connected to a master server at 127.5.77.254:34973
I20260924 12:48:19.261437 10428 consensus_queue.cc:260] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 } } peers { permanent_uuid: "e4fb0f890e4948bbaa5ffcfe269db2aa" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 34089 } } peers { permanent_uuid: "1dbbdaea267a40808a7cfc5ee97bcaf6" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 39413 } } peers { permanent_uuid: "3386972882594ac980b80f630606e76a" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35699 } } peers { permanent_uuid: "2f19a213b3d14397b2bd155d1984080b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 41513 } }
I20260924 12:48:19.261590 10537 heartbeater.cc:464] Registering TS with master...
I20260924 12:48:19.262308 10537 heartbeater.cc:511] Master 127.5.77.254:34973 requested a full tablet report, sending...
I20260924 12:48:19.263964 10428 ts_tablet_manager.cc:1428] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543: Time spent starting tablet: real 0.013s user 0.010s sys 0.001s
I20260924 12:48:19.264879 9401 ts_manager.cc:194] Re-registered known tserver with Master: 4834837931ea4b4db4517b8089ce7543 (127.5.77.194:38129)
I20260924 12:48:19.265934 9401 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.194:39527
I20260924 12:48:19.268584 10537 heartbeater.cc:503] Master 127.5.77.254:34973 was elected leader, sending a full tablet report...
I20260924 12:48:19.345768 10329 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260924 12:48:19.356025 9913 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260924 12:48:19.382256 10471 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260924 12:48:19.384430 9783 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260924 12:48:19.389678 9522 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
e46dcba8fbd2492b93dedc7222ac8f54 | 127.5.77.254:34973 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.77.212:45557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
1dbbdaea267a40808a7cfc5ee97bcaf6 | 127.5.77.197:39413 | HEALTHY | <none> | 0 | 0
2f19a213b3d14397b2bd155d1984080b | 127.5.77.196:41513 | HEALTHY | <none> | 0 | 0
3386972882594ac980b80f630606e76a | 127.5.77.195:35699 | HEALTHY | <none> | 1 | 0
4834837931ea4b4db4517b8089ce7543 | 127.5.77.194:38129 | HEALTHY | <none> | 0 | 0
e4fb0f890e4948bbaa5ffcfe269db2aa | 127.5.77.193:34089 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 5
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 5 server(s) checked
local_ip_for_outbound_sockets | 127.5.77.193 | experimental | 127.5.77.193:34089
local_ip_for_outbound_sockets | 127.5.77.194 | experimental | 127.5.77.194:38129
local_ip_for_outbound_sockets | 127.5.77.195 | experimental | 127.5.77.195:35699
local_ip_for_outbound_sockets | 127.5.77.196 | experimental | 127.5.77.196:41513
local_ip_for_outbound_sockets | 127.5.77.197 | experimental | 127.5.77.197:39413
never_fsync | true | unsafe,advanced | all 5 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 5 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 5 server(s) checked
server_dump_info_format | pb | hidden | all 5 server(s) checked
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.5.77.193:34089
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.77.194:38129
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.77.195:35699
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.5.77.196:41513
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.77.197:39413
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.77.212:45557 | all 5 server(s) checked
time_source | builtin | all 5 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 6 server(s) checked
Tablet Summary
Tablet e5419f6c60ca45c894a89c0a2ef92305 of table 'TestTable' is conflicted: 1 replicas' active configs disagree with the leader master's
4834837931ea4b4db4517b8089ce7543 (127.5.77.194:38129): RUNNING
e4fb0f890e4948bbaa5ffcfe269db2aa (127.5.77.193:34089): RUNNING
1dbbdaea267a40808a7cfc5ee97bcaf6 (127.5.77.197:39413): RUNNING
3386972882594ac980b80f630606e76a (127.5.77.195:35699): RUNNING [LEADER]
2f19a213b3d14397b2bd155d1984080b (127.5.77.196:41513): RUNNING
All reported replicas are:
A = 4834837931ea4b4db4517b8089ce7543
B = e4fb0f890e4948bbaa5ffcfe269db2aa
C = 1dbbdaea267a40808a7cfc5ee97bcaf6
D = 3386972882594ac980b80f630606e76a
E = 2f19a213b3d14397b2bd155d1984080b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------------------+--------------+--------------+------------
master | A B C D* E | | | Yes
A | A B C D E | 1 | -1 | Yes
B | A B C D* E | 4 | -1 | Yes
C | A B C D* E | 4 | -1 | Yes
D | A B C D* E | 4 | -1 | Yes
E | A B C D* E | 4 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+--------------------+---------------+---------+------------+------------------+-------------
TestTable | 5 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 5
Tables | 1
Tablets | 1
Replicas | 5
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260924 12:48:19.426949 5431 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260924 12:48:19.654129 10329 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260924 12:48:19.663939 10471 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260924 12:48:19.665249 9913 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260924 12:48:19.665592 9522 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260924 12:48:19.665654 9783 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
e46dcba8fbd2492b93dedc7222ac8f54 | 127.5.77.254:34973 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.77.212:45557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
1dbbdaea267a40808a7cfc5ee97bcaf6 | 127.5.77.197:39413 | HEALTHY | <none> | 0 | 0
2f19a213b3d14397b2bd155d1984080b | 127.5.77.196:41513 | HEALTHY | <none> | 0 | 0
3386972882594ac980b80f630606e76a | 127.5.77.195:35699 | HEALTHY | <none> | 1 | 0
4834837931ea4b4db4517b8089ce7543 | 127.5.77.194:38129 | HEALTHY | <none> | 0 | 0
e4fb0f890e4948bbaa5ffcfe269db2aa | 127.5.77.193:34089 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 5
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 5 server(s) checked
local_ip_for_outbound_sockets | 127.5.77.193 | experimental | 127.5.77.193:34089
local_ip_for_outbound_sockets | 127.5.77.194 | experimental | 127.5.77.194:38129
local_ip_for_outbound_sockets | 127.5.77.195 | experimental | 127.5.77.195:35699
local_ip_for_outbound_sockets | 127.5.77.196 | experimental | 127.5.77.196:41513
local_ip_for_outbound_sockets | 127.5.77.197 | experimental | 127.5.77.197:39413
never_fsync | true | unsafe,advanced | all 5 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 5 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 5 server(s) checked
server_dump_info_format | pb | hidden | all 5 server(s) checked
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.5.77.193:34089
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.77.194:38129
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.77.195:35699
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.5.77.196:41513
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.77.197:39413
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.77.212:45557 | all 5 server(s) checked
time_source | builtin | all 5 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 6 server(s) checked
Tablet Summary
Tablet e5419f6c60ca45c894a89c0a2ef92305 of table 'TestTable' is conflicted: 1 replicas' active configs disagree with the leader master's
4834837931ea4b4db4517b8089ce7543 (127.5.77.194:38129): RUNNING
e4fb0f890e4948bbaa5ffcfe269db2aa (127.5.77.193:34089): RUNNING
1dbbdaea267a40808a7cfc5ee97bcaf6 (127.5.77.197:39413): RUNNING
3386972882594ac980b80f630606e76a (127.5.77.195:35699): RUNNING [LEADER]
2f19a213b3d14397b2bd155d1984080b (127.5.77.196:41513): RUNNING
All reported replicas are:
A = 4834837931ea4b4db4517b8089ce7543
B = e4fb0f890e4948bbaa5ffcfe269db2aa
C = 1dbbdaea267a40808a7cfc5ee97bcaf6
D = 3386972882594ac980b80f630606e76a
E = 2f19a213b3d14397b2bd155d1984080b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------------------+--------------+--------------+------------
master | A B C D* E | | | Yes
A | A B C D E | 1 | -1 | Yes
B | A B C D* E | 4 | -1 | Yes
C | A B C D* E | 4 | -1 | Yes
D | A B C D* E | 4 | -1 | Yes
E | A B C D* E | 4 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+--------------------+---------------+---------+------------+------------------+-------------
TestTable | 5 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 5
Tables | 1
Tablets | 1
Replicas | 5
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260924 12:48:19.719550 5431 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260924 12:48:19.798900 10491 raft_consensus.cc:3037] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543 [term 1 FOLLOWER]: Advancing to term 4
I20260924 12:48:19.802977 10491 raft_consensus.cc:1261] T e5419f6c60ca45c894a89c0a2ef92305 P 4834837931ea4b4db4517b8089ce7543 [term 4 FOLLOWER]: Refusing update from remote peer 3386972882594ac980b80f630606e76a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20260924 12:48:19.803892 10576 consensus_queue.cc:1037] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "4834837931ea4b4db4517b8089ce7543" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 38129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 12:48:19.811686 10599 mvcc.cc:205] Tried to move back new op lower bound from 7332880770929745920 to 7332880770276384768. Current Snapshot: MvccSnapshot[applied={T|T < 7332880770775584768}]
I20260924 12:48:20.030441 9783 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260924 12:48:20.032137 10329 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260924 12:48:20.036697 9913 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260924 12:48:20.039781 10471 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260924 12:48:20.058712 9522 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
e46dcba8fbd2492b93dedc7222ac8f54 | 127.5.77.254:34973 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.77.212:45557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
1dbbdaea267a40808a7cfc5ee97bcaf6 | 127.5.77.197:39413 | HEALTHY | <none> | 0 | 0
2f19a213b3d14397b2bd155d1984080b | 127.5.77.196:41513 | HEALTHY | <none> | 0 | 0
3386972882594ac980b80f630606e76a | 127.5.77.195:35699 | HEALTHY | <none> | 1 | 0
4834837931ea4b4db4517b8089ce7543 | 127.5.77.194:38129 | HEALTHY | <none> | 0 | 0
e4fb0f890e4948bbaa5ffcfe269db2aa | 127.5.77.193:34089 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 5
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 5 server(s) checked
local_ip_for_outbound_sockets | 127.5.77.193 | experimental | 127.5.77.193:34089
local_ip_for_outbound_sockets | 127.5.77.194 | experimental | 127.5.77.194:38129
local_ip_for_outbound_sockets | 127.5.77.195 | experimental | 127.5.77.195:35699
local_ip_for_outbound_sockets | 127.5.77.196 | experimental | 127.5.77.196:41513
local_ip_for_outbound_sockets | 127.5.77.197 | experimental | 127.5.77.197:39413
never_fsync | true | unsafe,advanced | all 5 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 5 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 5 server(s) checked
server_dump_info_format | pb | hidden | all 5 server(s) checked
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.5.77.193:34089
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.77.194:38129
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.77.195:35699
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.5.77.196:41513
server_dump_info_path | /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.77.197:39413
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.77.212:45557 | all 5 server(s) checked
time_source | builtin | all 5 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 6 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 5 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 5
Tables | 1
Tablets | 1
Replicas | 5
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260924 12:48:20.094863 5431 log_verifier.cc:126] Checking tablet e5419f6c60ca45c894a89c0a2ef92305
I20260924 12:48:20.151145 5431 log_verifier.cc:177] Verified matching terms for 18 ops in tablet e5419f6c60ca45c894a89c0a2ef92305
I20260924 12:48:20.179607 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 9460
I20260924 12:48:20.180833 9582 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:20.297574 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 9460
I20260924 12:48:20.312630 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 10399
W20260924 12:48:20.312683 9387 connection.cc:535] server connection from 127.5.77.193:42241 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 12:48:20.319617 10531 generic_service.cc:196] Checking for leaks (request via RPC)
W20260924 12:48:20.376427 9738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.77.193:34089: connect: Connection refused (error 111) [suppressed 13 similar messages]
W20260924 12:48:20.379733 9738 consensus_peers.cc:487] T e5419f6c60ca45c894a89c0a2ef92305 P 3386972882594ac980b80f630606e76a -> Peer e4fb0f890e4948bbaa5ffcfe269db2aa (127.5.77.193:34089): Couldn't send request to peer e4fb0f890e4948bbaa5ffcfe269db2aa. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.193:34089: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 12:48:20.423246 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 10399
I20260924 12:48:20.442595 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 9721
I20260924 12:48:20.443661 9842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:20.566633 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 9721
I20260924 12:48:20.582583 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 9851
I20260924 12:48:20.583561 9973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:20.678241 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 9851
I20260924 12:48:20.691493 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 10266
I20260924 12:48:20.697414 10390 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:20.787352 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 10266
I20260924 12:48:20.800172 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 9370
I20260924 12:48:20.801187 9431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:20.913582 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 9370
2026-09-24T12:48:20Z chronyd exiting
[ OK ] RaftConsensusElectionITest.AutomaticLeaderElection (11501 ms)
[ RUN ] RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica
I20260924 12:48:20.955173 5431 test_util.cc:268] Using random seed: 62832651
I20260924 12:48:20.956790 5431 ts_itest-base.cc:112] Starting cluster with:
I20260924 12:48:20.956849 5431 ts_itest-base.cc:113] --------------
I20260924 12:48:20.956897 5431 ts_itest-base.cc:114] 1 tablet servers
I20260924 12:48:20.956941 5431 ts_itest-base.cc:115] 1 replicas per TS
I20260924 12:48:20.957000 5431 ts_itest-base.cc:116] --------------
2026-09-24T12:48:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:48:20Z Disabled control of system clock
I20260924 12:48:20.987365 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.77.254:42561
--webserver_interface=127.5.77.254
--webserver_port=0
--builtin_ntp_servers=127.5.77.212:39417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.77.254:42561 with env {}
W20260924 12:48:21.307401 10657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:21.307698 10657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:21.307812 10657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:21.315260 10657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:48:21.315363 10657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:21.315420 10657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:48:21.315495 10657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:48:21.325090 10657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:39417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.77.254:42561
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.77.254:42561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.77.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:21.326395 10657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:21.328161 10657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:21.339146 10662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:21.339176 10663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:21.339146 10665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:21.340118 10657 server_base.cc:1034] running on GCE node
I20260924 12:48:21.340838 10657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:21.342127 10657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:21.343336 10657 hybrid_clock.cc:648] HybridClock initialized: now 1790254101343282 us; error 43 us; skew 500 ppm
I20260924 12:48:21.343675 10657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:21.345816 10657 webserver.cc:460] Webserver started at http://127.5.77.254:33287/ using document root <none> and password file <none>
I20260924 12:48:21.346490 10657 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:21.346608 10657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:21.346894 10657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:21.349457 10657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "2b389be7c7ee4df2be47f91e95640bbe"
format_stamp: "Formatted at 2026-09-24 12:48:21 on dist-test-slave-b8q3"
I20260924 12:48:21.350189 10657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "2b389be7c7ee4df2be47f91e95640bbe"
format_stamp: "Formatted at 2026-09-24 12:48:21 on dist-test-slave-b8q3"
I20260924 12:48:21.355095 10657 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 12:48:21.358419 10671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:21.359764 10657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:48:21.359906 10657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2b389be7c7ee4df2be47f91e95640bbe"
format_stamp: "Formatted at 2026-09-24 12:48:21 on dist-test-slave-b8q3"
I20260924 12:48:21.360095 10657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:21.395502 10657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:21.396368 10657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:21.396643 10657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:21.413317 10657 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.254:42561
I20260924 12:48:21.413355 10722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.254:42561 every 8 connection(s)
I20260924 12:48:21.414705 10657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260924 12:48:21.416790 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 10657
I20260924 12:48:21.416937 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260924 12:48:21.419129 10723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:21.430299 10723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b389be7c7ee4df2be47f91e95640bbe: Bootstrap starting.
I20260924 12:48:21.433393 10723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b389be7c7ee4df2be47f91e95640bbe: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:21.434324 10723 log.cc:824] T 00000000000000000000000000000000 P 2b389be7c7ee4df2be47f91e95640bbe: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:21.436816 10723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b389be7c7ee4df2be47f91e95640bbe: No bootstrap required, opened a new log
I20260924 12:48:21.443673 10723 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b389be7c7ee4df2be47f91e95640bbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b389be7c7ee4df2be47f91e95640bbe" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 42561 } }
I20260924 12:48:21.444106 10723 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b389be7c7ee4df2be47f91e95640bbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:48:21.444206 10723 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b389be7c7ee4df2be47f91e95640bbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b389be7c7ee4df2be47f91e95640bbe, State: Initialized, Role: FOLLOWER
I20260924 12:48:21.444818 10723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b389be7c7ee4df2be47f91e95640bbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b389be7c7ee4df2be47f91e95640bbe" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 42561 } }
I20260924 12:48:21.445026 10723 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b389be7c7ee4df2be47f91e95640bbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:48:21.445151 10723 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b389be7c7ee4df2be47f91e95640bbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:48:21.445297 10723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b389be7c7ee4df2be47f91e95640bbe [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:21.447325 10723 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b389be7c7ee4df2be47f91e95640bbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b389be7c7ee4df2be47f91e95640bbe" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 42561 } }
I20260924 12:48:21.447795 10723 leader_election.cc:302] T 00000000000000000000000000000000 P 2b389be7c7ee4df2be47f91e95640bbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b389be7c7ee4df2be47f91e95640bbe; no voters:
I20260924 12:48:21.448356 10723 leader_election.cc:288] T 00000000000000000000000000000000 P 2b389be7c7ee4df2be47f91e95640bbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:48:21.448504 10728 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b389be7c7ee4df2be47f91e95640bbe [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:21.449160 10728 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b389be7c7ee4df2be47f91e95640bbe [term 1 LEADER]: Becoming Leader. State: Replica: 2b389be7c7ee4df2be47f91e95640bbe, State: Running, Role: LEADER
I20260924 12:48:21.449745 10728 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b389be7c7ee4df2be47f91e95640bbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b389be7c7ee4df2be47f91e95640bbe" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 42561 } }
I20260924 12:48:21.450263 10723 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b389be7c7ee4df2be47f91e95640bbe [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:48:21.452847 10730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b389be7c7ee4df2be47f91e95640bbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b389be7c7ee4df2be47f91e95640bbe. Latest consensus state: current_term: 1 leader_uuid: "2b389be7c7ee4df2be47f91e95640bbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b389be7c7ee4df2be47f91e95640bbe" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 42561 } } }
I20260924 12:48:21.453027 10729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b389be7c7ee4df2be47f91e95640bbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b389be7c7ee4df2be47f91e95640bbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b389be7c7ee4df2be47f91e95640bbe" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 42561 } } }
I20260924 12:48:21.453366 10729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b389be7c7ee4df2be47f91e95640bbe [sys.catalog]: This master's current role is: LEADER
I20260924 12:48:21.453250 10730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b389be7c7ee4df2be47f91e95640bbe [sys.catalog]: This master's current role is: LEADER
I20260924 12:48:21.455993 10738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:48:21.460068 10738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:48:21.468827 10738 catalog_manager.cc:1347] Generated new cluster ID: 3996974e8fcf4847af1dcce78291f95b
I20260924 12:48:21.468940 10738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:48:21.481092 10738 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:48:21.482631 10738 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:48:21.488292 10738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b389be7c7ee4df2be47f91e95640bbe: Generated new TSK 0
I20260924 12:48:21.489128 10738 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:48:21.501366 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.193:0
--local_ip_for_outbound_sockets=127.5.77.193
--webserver_interface=127.5.77.193
--webserver_port=0
--tserver_master_addrs=127.5.77.254:42561
--builtin_ntp_servers=127.5.77.212:39417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
W20260924 12:48:21.821805 10747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:21.822086 10747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:21.822189 10747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:21.829577 10747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:21.829774 10747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.193
I20260924 12:48:21.838923 10747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:39417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.77.193
--webserver_port=0
--tserver_master_addrs=127.5.77.254:42561
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.193
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:21.840283 10747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:21.841993 10747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:21.855224 10755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:21.855293 10752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:21.855295 10753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:21.856144 10747 server_base.cc:1034] running on GCE node
I20260924 12:48:21.856884 10747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:21.857951 10747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:21.859231 10747 hybrid_clock.cc:648] HybridClock initialized: now 1790254101859157 us; error 44 us; skew 500 ppm
I20260924 12:48:21.859526 10747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:21.861863 10747 webserver.cc:460] Webserver started at http://127.5.77.193:42529/ using document root <none> and password file <none>
I20260924 12:48:21.862586 10747 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:21.862694 10747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:21.863003 10747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:21.865972 10747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "01a58c4b2ca44a589e70ed96f1c66e93"
format_stamp: "Formatted at 2026-09-24 12:48:21 on dist-test-slave-b8q3"
I20260924 12:48:21.866765 10747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "01a58c4b2ca44a589e70ed96f1c66e93"
format_stamp: "Formatted at 2026-09-24 12:48:21 on dist-test-slave-b8q3"
I20260924 12:48:21.871944 10747 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 12:48:21.875480 10761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:21.876971 10747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 12:48:21.877128 10747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "01a58c4b2ca44a589e70ed96f1c66e93"
format_stamp: "Formatted at 2026-09-24 12:48:21 on dist-test-slave-b8q3"
I20260924 12:48:21.877345 10747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:21.892444 10747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:21.893262 10747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:21.893560 10747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:21.894492 10747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:21.895910 10747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:48:21.895978 10747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:21.896093 10747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:48:21.896145 10747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:21.924983 10747 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.193:42787
I20260924 12:48:21.925020 10873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.193:42787 every 8 connection(s)
I20260924 12:48:21.926252 10747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260924 12:48:21.931684 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 10747
I20260924 12:48:21.931895 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260924 12:48:21.940205 10874 heartbeater.cc:347] Connected to a master server at 127.5.77.254:42561
I20260924 12:48:21.940480 10874 heartbeater.cc:464] Registering TS with master...
I20260924 12:48:21.941169 10874 heartbeater.cc:511] Master 127.5.77.254:42561 requested a full tablet report, sending...
I20260924 12:48:21.943048 10688 ts_manager.cc:194] Registered new tserver with Master: 01a58c4b2ca44a589e70ed96f1c66e93 (127.5.77.193:42787)
I20260924 12:48:21.944468 10688 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.193:37921
I20260924 12:48:21.947599 5431 external_mini_cluster.cc:930] 1 TS(s) registered with all masters
I20260924 12:48:21.962414 10688 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59478:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20260924 12:48:21.990379 10809 tablet_service.cc:1467] Processing CreateTablet for tablet 8e10588e2981484caef4287672004429 (DEFAULT_TABLE table=TestTable [id=9c70881b00504bacbc2d1f5c6ad89c07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:48:21.991761 10809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e10588e2981484caef4287672004429. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:22.001956 10884 tablet_bootstrap.cc:492] T 8e10588e2981484caef4287672004429 P 01a58c4b2ca44a589e70ed96f1c66e93: Bootstrap starting.
I20260924 12:48:22.004843 10884 tablet_bootstrap.cc:654] T 8e10588e2981484caef4287672004429 P 01a58c4b2ca44a589e70ed96f1c66e93: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:22.005795 10884 log.cc:824] T 8e10588e2981484caef4287672004429 P 01a58c4b2ca44a589e70ed96f1c66e93: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:22.008229 10884 tablet_bootstrap.cc:492] T 8e10588e2981484caef4287672004429 P 01a58c4b2ca44a589e70ed96f1c66e93: No bootstrap required, opened a new log
I20260924 12:48:22.008497 10884 ts_tablet_manager.cc:1397] T 8e10588e2981484caef4287672004429 P 01a58c4b2ca44a589e70ed96f1c66e93: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 12:48:22.015497 10884 raft_consensus.cc:348] T 8e10588e2981484caef4287672004429 P 01a58c4b2ca44a589e70ed96f1c66e93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a58c4b2ca44a589e70ed96f1c66e93" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42787 } }
I20260924 12:48:22.015851 10884 raft_consensus.cc:374] T 8e10588e2981484caef4287672004429 P 01a58c4b2ca44a589e70ed96f1c66e93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:48:22.015921 10884 raft_consensus.cc:729] T 8e10588e2981484caef4287672004429 P 01a58c4b2ca44a589e70ed96f1c66e93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01a58c4b2ca44a589e70ed96f1c66e93, State: Initialized, Role: FOLLOWER
I20260924 12:48:22.016436 10884 consensus_queue.cc:260] T 8e10588e2981484caef4287672004429 P 01a58c4b2ca44a589e70ed96f1c66e93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a58c4b2ca44a589e70ed96f1c66e93" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42787 } }
I20260924 12:48:22.016633 10884 raft_consensus.cc:388] T 8e10588e2981484caef4287672004429 P 01a58c4b2ca44a589e70ed96f1c66e93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:48:22.016727 10884 raft_consensus.cc:482] T 8e10588e2981484caef4287672004429 P 01a58c4b2ca44a589e70ed96f1c66e93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:48:22.016847 10884 raft_consensus.cc:3037] T 8e10588e2981484caef4287672004429 P 01a58c4b2ca44a589e70ed96f1c66e93 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:22.018859 10884 raft_consensus.cc:504] T 8e10588e2981484caef4287672004429 P 01a58c4b2ca44a589e70ed96f1c66e93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a58c4b2ca44a589e70ed96f1c66e93" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42787 } }
I20260924 12:48:22.019311 10884 leader_election.cc:302] T 8e10588e2981484caef4287672004429 P 01a58c4b2ca44a589e70ed96f1c66e93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01a58c4b2ca44a589e70ed96f1c66e93; no voters:
I20260924 12:48:22.019846 10884 leader_election.cc:288] T 8e10588e2981484caef4287672004429 P 01a58c4b2ca44a589e70ed96f1c66e93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:48:22.019974 10886 raft_consensus.cc:2781] T 8e10588e2981484caef4287672004429 P 01a58c4b2ca44a589e70ed96f1c66e93 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:22.020630 10886 raft_consensus.cc:686] T 8e10588e2981484caef4287672004429 P 01a58c4b2ca44a589e70ed96f1c66e93 [term 1 LEADER]: Becoming Leader. State: Replica: 01a58c4b2ca44a589e70ed96f1c66e93, State: Running, Role: LEADER
I20260924 12:48:22.020887 10874 heartbeater.cc:503] Master 127.5.77.254:42561 was elected leader, sending a full tablet report...
I20260924 12:48:22.021271 10886 consensus_queue.cc:237] T 8e10588e2981484caef4287672004429 P 01a58c4b2ca44a589e70ed96f1c66e93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a58c4b2ca44a589e70ed96f1c66e93" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42787 } }
I20260924 12:48:22.021430 10884 ts_tablet_manager.cc:1428] T 8e10588e2981484caef4287672004429 P 01a58c4b2ca44a589e70ed96f1c66e93: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260924 12:48:22.026158 10688 catalog_manager.cc:5495] T 8e10588e2981484caef4287672004429 P 01a58c4b2ca44a589e70ed96f1c66e93 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01a58c4b2ca44a589e70ed96f1c66e93 (127.5.77.193). New cstate: current_term: 1 leader_uuid: "01a58c4b2ca44a589e70ed96f1c66e93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a58c4b2ca44a589e70ed96f1c66e93" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42787 } health_report { overall_health: HEALTHY } } }
I20260924 12:48:22.047770 5431 external_mini_cluster.cc:930] 1 TS(s) registered with all masters
I20260924 12:48:22.049885 5431 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 01a58c4b2ca44a589e70ed96f1c66e93 to finish bootstrapping
I20260924 12:48:22.067078 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 10747
I20260924 12:48:22.068141 10869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:22.153878 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 10747
I20260924 12:48:22.167204 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 10657
I20260924 12:48:22.168306 10718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:22.269347 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 10657
2026-09-24T12:48:22Z chronyd exiting
[ OK ] RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica (1347 ms)
[ RUN ] RaftConsensusElectionITest.LeaderStepDown
I20260924 12:48:22.302795 5431 test_util.cc:268] Using random seed: 64180258
I20260924 12:48:22.304343 5431 ts_itest-base.cc:112] Starting cluster with:
I20260924 12:48:22.304419 5431 ts_itest-base.cc:113] --------------
I20260924 12:48:22.304455 5431 ts_itest-base.cc:114] 3 tablet servers
I20260924 12:48:22.304500 5431 ts_itest-base.cc:115] 3 replicas per TS
I20260924 12:48:22.304533 5431 ts_itest-base.cc:116] --------------
2026-09-24T12:48:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:48:22Z Disabled control of system clock
I20260924 12:48:22.331429 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.77.254:38395
--webserver_interface=127.5.77.254
--webserver_port=0
--builtin_ntp_servers=127.5.77.212:46493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.77.254:38395
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20260924 12:48:22.643677 10902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:22.643992 10902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:22.644114 10902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:22.651471 10902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:48:22.651647 10902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:22.651726 10902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:48:22.651803 10902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:48:22.661386 10902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:46493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.5.77.254:38395
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.77.254:38395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.77.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:22.662789 10902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:22.664575 10902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:22.676081 10907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:22.676129 10910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:22.676146 10908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:22.676707 10902 server_base.cc:1034] running on GCE node
I20260924 12:48:22.677529 10902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:22.679044 10902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:22.680256 10902 hybrid_clock.cc:648] HybridClock initialized: now 1790254102680194 us; error 39 us; skew 500 ppm
I20260924 12:48:22.680578 10902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:22.682848 10902 webserver.cc:460] Webserver started at http://127.5.77.254:39591/ using document root <none> and password file <none>
I20260924 12:48:22.683531 10902 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:22.683635 10902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:22.683925 10902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:22.686606 10902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "6240142dba5242ee93fe3ba5f0d3c76c"
format_stamp: "Formatted at 2026-09-24 12:48:22 on dist-test-slave-b8q3"
I20260924 12:48:22.687363 10902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "6240142dba5242ee93fe3ba5f0d3c76c"
format_stamp: "Formatted at 2026-09-24 12:48:22 on dist-test-slave-b8q3"
I20260924 12:48:22.692480 10902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 12:48:22.696049 10916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:22.697506 10902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260924 12:48:22.697652 10902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
uuid: "6240142dba5242ee93fe3ba5f0d3c76c"
format_stamp: "Formatted at 2026-09-24 12:48:22 on dist-test-slave-b8q3"
I20260924 12:48:22.697831 10902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:22.721001 10902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:22.721856 10902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:22.722117 10902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:22.738866 10902 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.254:38395
I20260924 12:48:22.738888 10967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.254:38395 every 8 connection(s)
I20260924 12:48:22.740222 10902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260924 12:48:22.744184 10968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:22.748785 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 10902
I20260924 12:48:22.748998 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260924 12:48:22.755405 10968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6240142dba5242ee93fe3ba5f0d3c76c: Bootstrap starting.
I20260924 12:48:22.758415 10968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6240142dba5242ee93fe3ba5f0d3c76c: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:22.759464 10968 log.cc:824] T 00000000000000000000000000000000 P 6240142dba5242ee93fe3ba5f0d3c76c: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:22.762627 10968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6240142dba5242ee93fe3ba5f0d3c76c: No bootstrap required, opened a new log
I20260924 12:48:22.770310 10968 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6240142dba5242ee93fe3ba5f0d3c76c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6240142dba5242ee93fe3ba5f0d3c76c" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 38395 } }
I20260924 12:48:22.770709 10968 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6240142dba5242ee93fe3ba5f0d3c76c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:48:22.770797 10968 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6240142dba5242ee93fe3ba5f0d3c76c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6240142dba5242ee93fe3ba5f0d3c76c, State: Initialized, Role: FOLLOWER
I20260924 12:48:22.771373 10968 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6240142dba5242ee93fe3ba5f0d3c76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6240142dba5242ee93fe3ba5f0d3c76c" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 38395 } }
I20260924 12:48:22.771574 10968 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6240142dba5242ee93fe3ba5f0d3c76c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:48:22.771695 10968 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6240142dba5242ee93fe3ba5f0d3c76c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:48:22.771844 10968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6240142dba5242ee93fe3ba5f0d3c76c [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:22.773878 10968 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6240142dba5242ee93fe3ba5f0d3c76c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6240142dba5242ee93fe3ba5f0d3c76c" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 38395 } }
I20260924 12:48:22.774334 10968 leader_election.cc:302] T 00000000000000000000000000000000 P 6240142dba5242ee93fe3ba5f0d3c76c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6240142dba5242ee93fe3ba5f0d3c76c; no voters:
I20260924 12:48:22.774880 10968 leader_election.cc:288] T 00000000000000000000000000000000 P 6240142dba5242ee93fe3ba5f0d3c76c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:48:22.774997 10973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6240142dba5242ee93fe3ba5f0d3c76c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:22.775628 10973 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6240142dba5242ee93fe3ba5f0d3c76c [term 1 LEADER]: Becoming Leader. State: Replica: 6240142dba5242ee93fe3ba5f0d3c76c, State: Running, Role: LEADER
I20260924 12:48:22.776232 10973 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6240142dba5242ee93fe3ba5f0d3c76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6240142dba5242ee93fe3ba5f0d3c76c" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 38395 } }
I20260924 12:48:22.776712 10968 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6240142dba5242ee93fe3ba5f0d3c76c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:48:22.778869 10975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6240142dba5242ee93fe3ba5f0d3c76c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6240142dba5242ee93fe3ba5f0d3c76c. Latest consensus state: current_term: 1 leader_uuid: "6240142dba5242ee93fe3ba5f0d3c76c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6240142dba5242ee93fe3ba5f0d3c76c" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 38395 } } }
I20260924 12:48:22.779244 10975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6240142dba5242ee93fe3ba5f0d3c76c [sys.catalog]: This master's current role is: LEADER
I20260924 12:48:22.779204 10974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6240142dba5242ee93fe3ba5f0d3c76c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6240142dba5242ee93fe3ba5f0d3c76c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6240142dba5242ee93fe3ba5f0d3c76c" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 38395 } } }
I20260924 12:48:22.779534 10974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6240142dba5242ee93fe3ba5f0d3c76c [sys.catalog]: This master's current role is: LEADER
I20260924 12:48:22.781623 10978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:48:22.786804 10978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:48:22.795770 10978 catalog_manager.cc:1347] Generated new cluster ID: ffe8e45e18b14a18a8cdc5ab31ee9aff
I20260924 12:48:22.795954 10978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:48:22.807833 10978 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:48:22.809087 10978 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:48:22.818907 10978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6240142dba5242ee93fe3ba5f0d3c76c: Generated new TSK 0
I20260924 12:48:22.819763 10978 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:48:22.829032 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.193:0
--local_ip_for_outbound_sockets=127.5.77.193
--webserver_interface=127.5.77.193
--webserver_port=0
--tserver_master_addrs=127.5.77.254:38395
--builtin_ntp_servers=127.5.77.212:46493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20260924 12:48:23.170024 10992 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 12:48:23.170359 10992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:23.170492 10992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:23.170572 10992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:23.177820 10992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:23.177989 10992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.193
I20260924 12:48:23.187350 10992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:46493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.77.193
--webserver_port=0
--tserver_master_addrs=127.5.77.254:38395
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.193
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:23.188592 10992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:23.190250 10992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:23.203040 10998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:23.203012 11000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:23.203081 10997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:23.203745 10992 server_base.cc:1034] running on GCE node
I20260924 12:48:23.204375 10992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:23.205318 10992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:23.206550 10992 hybrid_clock.cc:648] HybridClock initialized: now 1790254103206488 us; error 48 us; skew 500 ppm
I20260924 12:48:23.206879 10992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:23.209049 10992 webserver.cc:460] Webserver started at http://127.5.77.193:40891/ using document root <none> and password file <none>
I20260924 12:48:23.209724 10992 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:23.209828 10992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:23.210139 10992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:23.212862 10992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "80b5c8422f21490585082fb94bd560e9"
format_stamp: "Formatted at 2026-09-24 12:48:23 on dist-test-slave-b8q3"
I20260924 12:48:23.213584 10992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "80b5c8422f21490585082fb94bd560e9"
format_stamp: "Formatted at 2026-09-24 12:48:23 on dist-test-slave-b8q3"
I20260924 12:48:23.218557 10992 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260924 12:48:23.222083 11006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:23.223462 10992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 12:48:23.223613 10992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "80b5c8422f21490585082fb94bd560e9"
format_stamp: "Formatted at 2026-09-24 12:48:23 on dist-test-slave-b8q3"
I20260924 12:48:23.223801 10992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:23.244943 10992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:23.245874 10992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:23.246214 10992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:23.247231 10992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:23.248718 10992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:48:23.248822 10992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:23.248946 10992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:48:23.249022 10992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:23.278949 10992 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.193:46277
I20260924 12:48:23.278997 11118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.193:46277 every 8 connection(s)
I20260924 12:48:23.280236 10992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260924 12:48:23.282491 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 10992
I20260924 12:48:23.282655 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260924 12:48:23.285760 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.194:0
--local_ip_for_outbound_sockets=127.5.77.194
--webserver_interface=127.5.77.194
--webserver_port=0
--tserver_master_addrs=127.5.77.254:38395
--builtin_ntp_servers=127.5.77.212:46493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260924 12:48:23.293920 11119 heartbeater.cc:347] Connected to a master server at 127.5.77.254:38395
I20260924 12:48:23.294256 11119 heartbeater.cc:464] Registering TS with master...
I20260924 12:48:23.295130 11119 heartbeater.cc:511] Master 127.5.77.254:38395 requested a full tablet report, sending...
I20260924 12:48:23.297174 10933 ts_manager.cc:194] Registered new tserver with Master: 80b5c8422f21490585082fb94bd560e9 (127.5.77.193:46277)
I20260924 12:48:23.298686 10933 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.193:36859
W20260924 12:48:23.600330 11123 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 12:48:23.600679 11123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:23.600767 11123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:23.600864 11123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:23.608369 11123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:23.608551 11123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.194
I20260924 12:48:23.617841 11123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:46493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.77.194
--webserver_port=0
--tserver_master_addrs=127.5.77.254:38395
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.194
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:23.619196 11123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:23.620965 11123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:23.633766 11128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:23.633736 11129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:23.633848 11131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:23.634738 11123 server_base.cc:1034] running on GCE node
I20260924 12:48:23.635341 11123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:23.636360 11123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:23.637570 11123 hybrid_clock.cc:648] HybridClock initialized: now 1790254103637524 us; error 44 us; skew 500 ppm
I20260924 12:48:23.637912 11123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:23.640249 11123 webserver.cc:460] Webserver started at http://127.5.77.194:43647/ using document root <none> and password file <none>
I20260924 12:48:23.640976 11123 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:23.641070 11123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:23.641374 11123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:23.644049 11123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "b4d57cdabf124987a5fcef4c928491ac"
format_stamp: "Formatted at 2026-09-24 12:48:23 on dist-test-slave-b8q3"
I20260924 12:48:23.644833 11123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "b4d57cdabf124987a5fcef4c928491ac"
format_stamp: "Formatted at 2026-09-24 12:48:23 on dist-test-slave-b8q3"
I20260924 12:48:23.650003 11123 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 12:48:23.653617 11137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:23.655049 11123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:48:23.655202 11123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "b4d57cdabf124987a5fcef4c928491ac"
format_stamp: "Formatted at 2026-09-24 12:48:23 on dist-test-slave-b8q3"
I20260924 12:48:23.655413 11123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:23.669471 11123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:23.670329 11123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:23.670636 11123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:23.671633 11123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:23.673244 11123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:48:23.673347 11123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:23.673453 11123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:48:23.673518 11123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:23.703356 11123 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.194:35013
I20260924 12:48:23.703421 11249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.194:35013 every 8 connection(s)
I20260924 12:48:23.704785 11123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260924 12:48:23.706350 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 11123
I20260924 12:48:23.706514 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260924 12:48:23.709502 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.195:0
--local_ip_for_outbound_sockets=127.5.77.195
--webserver_interface=127.5.77.195
--webserver_port=0
--tserver_master_addrs=127.5.77.254:38395
--builtin_ntp_servers=127.5.77.212:46493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260924 12:48:23.722199 11250 heartbeater.cc:347] Connected to a master server at 127.5.77.254:38395
I20260924 12:48:23.722579 11250 heartbeater.cc:464] Registering TS with master...
I20260924 12:48:23.723369 11250 heartbeater.cc:511] Master 127.5.77.254:38395 requested a full tablet report, sending...
I20260924 12:48:23.724861 10933 ts_manager.cc:194] Registered new tserver with Master: b4d57cdabf124987a5fcef4c928491ac (127.5.77.194:35013)
I20260924 12:48:23.726078 10933 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.194:48091
W20260924 12:48:24.031159 11253 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 12:48:24.031468 11253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:24.031526 11253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:24.031596 11253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:24.039042 11253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:24.039203 11253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.195
I20260924 12:48:24.048290 11253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:46493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.77.195
--webserver_port=0
--tserver_master_addrs=127.5.77.254:38395
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.195
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:24.049606 11253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:24.051309 11253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:24.063771 11261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:24.063750 11259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:24.063772 11258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:24.064273 11253 server_base.cc:1034] running on GCE node
I20260924 12:48:24.065047 11253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:24.065991 11253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:24.067227 11253 hybrid_clock.cc:648] HybridClock initialized: now 1790254104067172 us; error 49 us; skew 500 ppm
I20260924 12:48:24.067565 11253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:24.069873 11253 webserver.cc:460] Webserver started at http://127.5.77.195:36867/ using document root <none> and password file <none>
I20260924 12:48:24.070665 11253 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:24.070781 11253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:24.071075 11253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:24.073812 11253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "53e61ffeb4cd454fb3b8ccda7c2bef3b"
format_stamp: "Formatted at 2026-09-24 12:48:24 on dist-test-slave-b8q3"
I20260924 12:48:24.074633 11253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "53e61ffeb4cd454fb3b8ccda7c2bef3b"
format_stamp: "Formatted at 2026-09-24 12:48:24 on dist-test-slave-b8q3"
I20260924 12:48:24.079973 11253 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 12:48:24.083630 11267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:24.085062 11253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 12:48:24.085223 11253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "53e61ffeb4cd454fb3b8ccda7c2bef3b"
format_stamp: "Formatted at 2026-09-24 12:48:24 on dist-test-slave-b8q3"
I20260924 12:48:24.085433 11253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:24.106093 11253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:24.107012 11253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:24.107298 11253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:24.108346 11253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:24.109961 11253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:48:24.110052 11253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:24.110204 11253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:48:24.110286 11253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:24.139876 11253 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.195:40151
I20260924 12:48:24.139914 11379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.195:40151 every 8 connection(s)
I20260924 12:48:24.141179 11253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260924 12:48:24.150503 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 11253
I20260924 12:48:24.150674 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260924 12:48:24.154554 11380 heartbeater.cc:347] Connected to a master server at 127.5.77.254:38395
I20260924 12:48:24.154837 11380 heartbeater.cc:464] Registering TS with master...
I20260924 12:48:24.155514 11380 heartbeater.cc:511] Master 127.5.77.254:38395 requested a full tablet report, sending...
I20260924 12:48:24.156787 10933 ts_manager.cc:194] Registered new tserver with Master: 53e61ffeb4cd454fb3b8ccda7c2bef3b (127.5.77.195:40151)
I20260924 12:48:24.157495 10933 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.195:46817
I20260924 12:48:24.166435 5431 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:48:24.181214 10933 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59428:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20260924 12:48:24.187812 10933 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:48:24.218278 11185 tablet_service.cc:1467] Processing CreateTablet for tablet 2f324d7e3d6740259480d9cbbfe6c0b1 (DEFAULT_TABLE table=TestTable [id=7f71deedfe4542b7a5402133edd4c3b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:48:24.220145 11185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f324d7e3d6740259480d9cbbfe6c0b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:24.222913 11315 tablet_service.cc:1467] Processing CreateTablet for tablet 2f324d7e3d6740259480d9cbbfe6c0b1 (DEFAULT_TABLE table=TestTable [id=7f71deedfe4542b7a5402133edd4c3b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:48:24.223469 11054 tablet_service.cc:1467] Processing CreateTablet for tablet 2f324d7e3d6740259480d9cbbfe6c0b1 (DEFAULT_TABLE table=TestTable [id=7f71deedfe4542b7a5402133edd4c3b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:48:24.224103 11315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f324d7e3d6740259480d9cbbfe6c0b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:24.224761 11054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f324d7e3d6740259480d9cbbfe6c0b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:24.233916 11398 tablet_bootstrap.cc:492] T 2f324d7e3d6740259480d9cbbfe6c0b1 P b4d57cdabf124987a5fcef4c928491ac: Bootstrap starting.
I20260924 12:48:24.237284 11400 tablet_bootstrap.cc:492] T 2f324d7e3d6740259480d9cbbfe6c0b1 P 80b5c8422f21490585082fb94bd560e9: Bootstrap starting.
I20260924 12:48:24.238793 11399 tablet_bootstrap.cc:492] T 2f324d7e3d6740259480d9cbbfe6c0b1 P 53e61ffeb4cd454fb3b8ccda7c2bef3b: Bootstrap starting.
I20260924 12:48:24.239409 11398 tablet_bootstrap.cc:654] T 2f324d7e3d6740259480d9cbbfe6c0b1 P b4d57cdabf124987a5fcef4c928491ac: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:24.240455 11398 log.cc:824] T 2f324d7e3d6740259480d9cbbfe6c0b1 P b4d57cdabf124987a5fcef4c928491ac: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:24.241098 11400 tablet_bootstrap.cc:654] T 2f324d7e3d6740259480d9cbbfe6c0b1 P 80b5c8422f21490585082fb94bd560e9: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:24.242035 11399 tablet_bootstrap.cc:654] T 2f324d7e3d6740259480d9cbbfe6c0b1 P 53e61ffeb4cd454fb3b8ccda7c2bef3b: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:24.242231 11400 log.cc:824] T 2f324d7e3d6740259480d9cbbfe6c0b1 P 80b5c8422f21490585082fb94bd560e9: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:24.242877 11398 tablet_bootstrap.cc:492] T 2f324d7e3d6740259480d9cbbfe6c0b1 P b4d57cdabf124987a5fcef4c928491ac: No bootstrap required, opened a new log
I20260924 12:48:24.243132 11399 log.cc:824] T 2f324d7e3d6740259480d9cbbfe6c0b1 P 53e61ffeb4cd454fb3b8ccda7c2bef3b: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:24.243152 11398 ts_tablet_manager.cc:1397] T 2f324d7e3d6740259480d9cbbfe6c0b1 P b4d57cdabf124987a5fcef4c928491ac: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 12:48:24.244966 11400 tablet_bootstrap.cc:492] T 2f324d7e3d6740259480d9cbbfe6c0b1 P 80b5c8422f21490585082fb94bd560e9: No bootstrap required, opened a new log
I20260924 12:48:24.245239 11400 ts_tablet_manager.cc:1397] T 2f324d7e3d6740259480d9cbbfe6c0b1 P 80b5c8422f21490585082fb94bd560e9: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260924 12:48:24.245781 11399 tablet_bootstrap.cc:492] T 2f324d7e3d6740259480d9cbbfe6c0b1 P 53e61ffeb4cd454fb3b8ccda7c2bef3b: No bootstrap required, opened a new log
I20260924 12:48:24.246021 11399 ts_tablet_manager.cc:1397] T 2f324d7e3d6740259480d9cbbfe6c0b1 P 53e61ffeb4cd454fb3b8ccda7c2bef3b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260924 12:48:24.250864 11398 raft_consensus.cc:348] T 2f324d7e3d6740259480d9cbbfe6c0b1 P b4d57cdabf124987a5fcef4c928491ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4d57cdabf124987a5fcef4c928491ac" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35013 } } peers { permanent_uuid: "80b5c8422f21490585082fb94bd560e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46277 } } peers { permanent_uuid: "53e61ffeb4cd454fb3b8ccda7c2bef3b" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 40151 } }
I20260924 12:48:24.251222 11398 raft_consensus.cc:729] T 2f324d7e3d6740259480d9cbbfe6c0b1 P b4d57cdabf124987a5fcef4c928491ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4d57cdabf124987a5fcef4c928491ac, State: Initialized, Role: FOLLOWER
I20260924 12:48:24.251668 11398 consensus_queue.cc:260] T 2f324d7e3d6740259480d9cbbfe6c0b1 P b4d57cdabf124987a5fcef4c928491ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4d57cdabf124987a5fcef4c928491ac" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35013 } } peers { permanent_uuid: "80b5c8422f21490585082fb94bd560e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46277 } } peers { permanent_uuid: "53e61ffeb4cd454fb3b8ccda7c2bef3b" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 40151 } }
I20260924 12:48:24.252697 11250 heartbeater.cc:503] Master 127.5.77.254:38395 was elected leader, sending a full tablet report...
I20260924 12:48:24.255021 11398 ts_tablet_manager.cc:1428] T 2f324d7e3d6740259480d9cbbfe6c0b1 P b4d57cdabf124987a5fcef4c928491ac: Time spent starting tablet: real 0.012s user 0.005s sys 0.004s
I20260924 12:48:24.255312 11400 raft_consensus.cc:348] T 2f324d7e3d6740259480d9cbbfe6c0b1 P 80b5c8422f21490585082fb94bd560e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4d57cdabf124987a5fcef4c928491ac" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35013 } } peers { permanent_uuid: "80b5c8422f21490585082fb94bd560e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46277 } } peers { permanent_uuid: "53e61ffeb4cd454fb3b8ccda7c2bef3b" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 40151 } }
I20260924 12:48:24.255759 11400 raft_consensus.cc:729] T 2f324d7e3d6740259480d9cbbfe6c0b1 P 80b5c8422f21490585082fb94bd560e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80b5c8422f21490585082fb94bd560e9, State: Initialized, Role: FOLLOWER
I20260924 12:48:24.255915 11399 raft_consensus.cc:348] T 2f324d7e3d6740259480d9cbbfe6c0b1 P 53e61ffeb4cd454fb3b8ccda7c2bef3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4d57cdabf124987a5fcef4c928491ac" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35013 } } peers { permanent_uuid: "80b5c8422f21490585082fb94bd560e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46277 } } peers { permanent_uuid: "53e61ffeb4cd454fb3b8ccda7c2bef3b" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 40151 } }
I20260924 12:48:24.256325 11399 raft_consensus.cc:729] T 2f324d7e3d6740259480d9cbbfe6c0b1 P 53e61ffeb4cd454fb3b8ccda7c2bef3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e61ffeb4cd454fb3b8ccda7c2bef3b, State: Initialized, Role: FOLLOWER
I20260924 12:48:24.256222 11400 consensus_queue.cc:260] T 2f324d7e3d6740259480d9cbbfe6c0b1 P 80b5c8422f21490585082fb94bd560e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4d57cdabf124987a5fcef4c928491ac" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35013 } } peers { permanent_uuid: "80b5c8422f21490585082fb94bd560e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46277 } } peers { permanent_uuid: "53e61ffeb4cd454fb3b8ccda7c2bef3b" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 40151 } }
I20260924 12:48:24.256747 11399 consensus_queue.cc:260] T 2f324d7e3d6740259480d9cbbfe6c0b1 P 53e61ffeb4cd454fb3b8ccda7c2bef3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4d57cdabf124987a5fcef4c928491ac" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35013 } } peers { permanent_uuid: "80b5c8422f21490585082fb94bd560e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46277 } } peers { permanent_uuid: "53e61ffeb4cd454fb3b8ccda7c2bef3b" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 40151 } }
I20260924 12:48:24.257961 11400 ts_tablet_manager.cc:1428] T 2f324d7e3d6740259480d9cbbfe6c0b1 P 80b5c8422f21490585082fb94bd560e9: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260924 12:48:24.258440 11119 heartbeater.cc:503] Master 127.5.77.254:38395 was elected leader, sending a full tablet report...
I20260924 12:48:24.258452 11399 ts_tablet_manager.cc:1428] T 2f324d7e3d6740259480d9cbbfe6c0b1 P 53e61ffeb4cd454fb3b8ccda7c2bef3b: Time spent starting tablet: real 0.012s user 0.008s sys 0.006s
I20260924 12:48:24.259645 11380 heartbeater.cc:503] Master 127.5.77.254:38395 was elected leader, sending a full tablet report...
I20260924 12:48:24.267590 5431 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:48:24.269598 5431 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 80b5c8422f21490585082fb94bd560e9 to finish bootstrapping
I20260924 12:48:24.278522 5431 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b4d57cdabf124987a5fcef4c928491ac to finish bootstrapping
W20260924 12:48:24.282603 11120 tablet.cc:2369] T 2f324d7e3d6740259480d9cbbfe6c0b1 P 80b5c8422f21490585082fb94bd560e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:48:24.287182 5431 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 53e61ffeb4cd454fb3b8ccda7c2bef3b to finish bootstrapping
I20260924 12:48:24.304296 11205 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2f324d7e3d6740259480d9cbbfe6c0b1"
dest_uuid: "b4d57cdabf124987a5fcef4c928491ac"
from {username='slave'} at 127.0.0.1:38702
I20260924 12:48:24.304608 11205 raft_consensus.cc:482] T 2f324d7e3d6740259480d9cbbfe6c0b1 P b4d57cdabf124987a5fcef4c928491ac [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 12:48:24.304759 11205 raft_consensus.cc:3037] T 2f324d7e3d6740259480d9cbbfe6c0b1 P b4d57cdabf124987a5fcef4c928491ac [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:24.306869 11205 raft_consensus.cc:504] T 2f324d7e3d6740259480d9cbbfe6c0b1 P b4d57cdabf124987a5fcef4c928491ac [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4d57cdabf124987a5fcef4c928491ac" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35013 } } peers { permanent_uuid: "80b5c8422f21490585082fb94bd560e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46277 } } peers { permanent_uuid: "53e61ffeb4cd454fb3b8ccda7c2bef3b" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 40151 } }
I20260924 12:48:24.308362 11205 leader_election.cc:288] T 2f324d7e3d6740259480d9cbbfe6c0b1 P b4d57cdabf124987a5fcef4c928491ac [CANDIDATE]: Term 1 election: Requested vote from peers 80b5c8422f21490585082fb94bd560e9 (127.5.77.193:46277), 53e61ffeb4cd454fb3b8ccda7c2bef3b (127.5.77.195:40151)
I20260924 12:48:24.317327 11074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f324d7e3d6740259480d9cbbfe6c0b1" candidate_uuid: "b4d57cdabf124987a5fcef4c928491ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "80b5c8422f21490585082fb94bd560e9"
I20260924 12:48:24.317338 11335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f324d7e3d6740259480d9cbbfe6c0b1" candidate_uuid: "b4d57cdabf124987a5fcef4c928491ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "53e61ffeb4cd454fb3b8ccda7c2bef3b"
I20260924 12:48:24.317700 11074 raft_consensus.cc:3037] T 2f324d7e3d6740259480d9cbbfe6c0b1 P 80b5c8422f21490585082fb94bd560e9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:24.317698 11335 raft_consensus.cc:3037] T 2f324d7e3d6740259480d9cbbfe6c0b1 P 53e61ffeb4cd454fb3b8ccda7c2bef3b [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:24.319805 11074 raft_consensus.cc:2454] T 2f324d7e3d6740259480d9cbbfe6c0b1 P 80b5c8422f21490585082fb94bd560e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4d57cdabf124987a5fcef4c928491ac in term 1.
I20260924 12:48:24.319813 11335 raft_consensus.cc:2454] T 2f324d7e3d6740259480d9cbbfe6c0b1 P 53e61ffeb4cd454fb3b8ccda7c2bef3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4d57cdabf124987a5fcef4c928491ac in term 1.
I20260924 12:48:24.320370 11141 leader_election.cc:302] T 2f324d7e3d6740259480d9cbbfe6c0b1 P b4d57cdabf124987a5fcef4c928491ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80b5c8422f21490585082fb94bd560e9, b4d57cdabf124987a5fcef4c928491ac; no voters:
I20260924 12:48:24.320822 11404 raft_consensus.cc:2781] T 2f324d7e3d6740259480d9cbbfe6c0b1 P b4d57cdabf124987a5fcef4c928491ac [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:24.321403 11404 raft_consensus.cc:686] T 2f324d7e3d6740259480d9cbbfe6c0b1 P b4d57cdabf124987a5fcef4c928491ac [term 1 LEADER]: Becoming Leader. State: Replica: b4d57cdabf124987a5fcef4c928491ac, State: Running, Role: LEADER
I20260924 12:48:24.321887 11404 consensus_queue.cc:237] T 2f324d7e3d6740259480d9cbbfe6c0b1 P b4d57cdabf124987a5fcef4c928491ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4d57cdabf124987a5fcef4c928491ac" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35013 } } peers { permanent_uuid: "80b5c8422f21490585082fb94bd560e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46277 } } peers { permanent_uuid: "53e61ffeb4cd454fb3b8ccda7c2bef3b" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 40151 } }
I20260924 12:48:24.326200 10933 catalog_manager.cc:5495] T 2f324d7e3d6740259480d9cbbfe6c0b1 P b4d57cdabf124987a5fcef4c928491ac reported cstate change: term changed from 0 to 1, leader changed from <none> to b4d57cdabf124987a5fcef4c928491ac (127.5.77.194). New cstate: current_term: 1 leader_uuid: "b4d57cdabf124987a5fcef4c928491ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4d57cdabf124987a5fcef4c928491ac" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80b5c8422f21490585082fb94bd560e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53e61ffeb4cd454fb3b8ccda7c2bef3b" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 40151 } health_report { overall_health: UNKNOWN } } }
I20260924 12:48:24.337476 11335 raft_consensus.cc:1261] T 2f324d7e3d6740259480d9cbbfe6c0b1 P 53e61ffeb4cd454fb3b8ccda7c2bef3b [term 1 FOLLOWER]: Refusing update from remote peer b4d57cdabf124987a5fcef4c928491ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:48:24.337484 11074 raft_consensus.cc:1261] T 2f324d7e3d6740259480d9cbbfe6c0b1 P 80b5c8422f21490585082fb94bd560e9 [term 1 FOLLOWER]: Refusing update from remote peer b4d57cdabf124987a5fcef4c928491ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:48:24.338333 11411 consensus_queue.cc:1035] T 2f324d7e3d6740259480d9cbbfe6c0b1 P b4d57cdabf124987a5fcef4c928491ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e61ffeb4cd454fb3b8ccda7c2bef3b" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 40151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:48:24.338778 11404 consensus_queue.cc:1035] T 2f324d7e3d6740259480d9cbbfe6c0b1 P b4d57cdabf124987a5fcef4c928491ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "80b5c8422f21490585082fb94bd560e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:48:24.348173 11413 mvcc.cc:205] Tried to move back new op lower bound from 7332880811357851648 to 7332880811306954752. Current Snapshot: MvccSnapshot[applied={T|T < 7332880811357851648}]
I20260924 12:48:24.370829 11205 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2f324d7e3d6740259480d9cbbfe6c0b1"
dest_uuid: "b4d57cdabf124987a5fcef4c928491ac"
from {username='slave'} at 127.0.0.1:38702
I20260924 12:48:24.371110 11205 raft_consensus.cc:580] T 2f324d7e3d6740259480d9cbbfe6c0b1 P b4d57cdabf124987a5fcef4c928491ac [term 1 LEADER]: Received request to step down
I20260924 12:48:24.371186 11205 raft_consensus.cc:3032] T 2f324d7e3d6740259480d9cbbfe6c0b1 P b4d57cdabf124987a5fcef4c928491ac [term 1 LEADER]: Stepping down as leader of term 1
I20260924 12:48:24.371244 11205 raft_consensus.cc:729] T 2f324d7e3d6740259480d9cbbfe6c0b1 P b4d57cdabf124987a5fcef4c928491ac [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b4d57cdabf124987a5fcef4c928491ac, State: Running, Role: LEADER
I20260924 12:48:24.371443 11205 consensus_queue.cc:260] T 2f324d7e3d6740259480d9cbbfe6c0b1 P b4d57cdabf124987a5fcef4c928491ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4d57cdabf124987a5fcef4c928491ac" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 35013 } } peers { permanent_uuid: "80b5c8422f21490585082fb94bd560e9" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46277 } } peers { permanent_uuid: "53e61ffeb4cd454fb3b8ccda7c2bef3b" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 40151 } }
I20260924 12:48:24.371932 11205 raft_consensus.cc:3037] T 2f324d7e3d6740259480d9cbbfe6c0b1 P b4d57cdabf124987a5fcef4c928491ac [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:48:24.373874 11205 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2f324d7e3d6740259480d9cbbfe6c0b1"
dest_uuid: "b4d57cdabf124987a5fcef4c928491ac"
from {username='slave'} at 127.0.0.1:38702
I20260924 12:48:24.374065 11205 raft_consensus.cc:573] T 2f324d7e3d6740259480d9cbbfe6c0b1 P b4d57cdabf124987a5fcef4c928491ac [term 2 FOLLOWER]: Rejecting request to step down while not leader
W20260924 12:48:24.375957 11165 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38702: Illegal state: replica b4d57cdabf124987a5fcef4c928491ac is not leader of this config: current role FOLLOWER
I20260924 12:48:24.377167 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 10992
I20260924 12:48:24.378429 11114 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:24.466259 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 10992
I20260924 12:48:24.479869 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 11123
I20260924 12:48:24.480888 11245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:24.571421 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 11123
I20260924 12:48:24.584806 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 11253
I20260924 12:48:24.585919 11375 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:24.674620 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 11253
I20260924 12:48:24.688068 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 10902
I20260924 12:48:24.689138 10963 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:24.793145 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 10902
2026-09-24T12:48:24Z chronyd exiting
[ OK ] RaftConsensusElectionITest.LeaderStepDown (2525 ms)
[ RUN ] RaftConsensusElectionITest.StepDownWithSlowFollower
I20260924 12:48:24.828819 5431 test_util.cc:268] Using random seed: 66706283
I20260924 12:48:24.830510 5431 ts_itest-base.cc:112] Starting cluster with:
I20260924 12:48:24.830579 5431 ts_itest-base.cc:113] --------------
I20260924 12:48:24.830621 5431 ts_itest-base.cc:114] 3 tablet servers
I20260924 12:48:24.830650 5431 ts_itest-base.cc:115] 3 replicas per TS
I20260924 12:48:24.830695 5431 ts_itest-base.cc:116] --------------
2026-09-24T12:48:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:48:24Z Disabled control of system clock
I20260924 12:48:24.860934 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.77.254:39827
--webserver_interface=127.5.77.254
--webserver_port=0
--builtin_ntp_servers=127.5.77.212:44827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.77.254:39827
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20260924 12:48:25.185310 11433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:25.185678 11433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:25.185813 11433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:25.193315 11433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:48:25.193477 11433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:25.193557 11433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:48:25.193653 11433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:48:25.203370 11433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:44827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.5.77.254:39827
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.77.254:39827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.77.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:25.204838 11433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:25.206740 11433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:25.219038 11441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:25.219149 11438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:25.219090 11439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:25.219751 11433 server_base.cc:1034] running on GCE node
I20260924 12:48:25.220551 11433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:25.221885 11433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:25.223086 11433 hybrid_clock.cc:648] HybridClock initialized: now 1790254105223010 us; error 57 us; skew 500 ppm
I20260924 12:48:25.223404 11433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:25.225669 11433 webserver.cc:460] Webserver started at http://127.5.77.254:44631/ using document root <none> and password file <none>
I20260924 12:48:25.226327 11433 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:25.226444 11433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:25.226766 11433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:25.229590 11433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "3f9fab5de21c4290920a69f4d73c3ef2"
format_stamp: "Formatted at 2026-09-24 12:48:25 on dist-test-slave-b8q3"
I20260924 12:48:25.230371 11433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "3f9fab5de21c4290920a69f4d73c3ef2"
format_stamp: "Formatted at 2026-09-24 12:48:25 on dist-test-slave-b8q3"
I20260924 12:48:25.235608 11433 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 12:48:25.238881 11447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:25.240244 11433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 12:48:25.240418 11433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3f9fab5de21c4290920a69f4d73c3ef2"
format_stamp: "Formatted at 2026-09-24 12:48:25 on dist-test-slave-b8q3"
I20260924 12:48:25.240649 11433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:25.269021 11433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:25.269937 11433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:25.270229 11433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:25.287371 11433 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.254:39827
I20260924 12:48:25.287396 11498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.254:39827 every 8 connection(s)
I20260924 12:48:25.288704 11433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260924 12:48:25.291606 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 11433
I20260924 12:48:25.291790 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260924 12:48:25.293044 11499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:25.305888 11499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f9fab5de21c4290920a69f4d73c3ef2: Bootstrap starting.
I20260924 12:48:25.308872 11499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f9fab5de21c4290920a69f4d73c3ef2: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:25.309845 11499 log.cc:824] T 00000000000000000000000000000000 P 3f9fab5de21c4290920a69f4d73c3ef2: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:25.312332 11499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f9fab5de21c4290920a69f4d73c3ef2: No bootstrap required, opened a new log
I20260924 12:48:25.319413 11499 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f9fab5de21c4290920a69f4d73c3ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9fab5de21c4290920a69f4d73c3ef2" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 39827 } }
I20260924 12:48:25.319835 11499 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f9fab5de21c4290920a69f4d73c3ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:48:25.319947 11499 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f9fab5de21c4290920a69f4d73c3ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f9fab5de21c4290920a69f4d73c3ef2, State: Initialized, Role: FOLLOWER
I20260924 12:48:25.320583 11499 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f9fab5de21c4290920a69f4d73c3ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9fab5de21c4290920a69f4d73c3ef2" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 39827 } }
I20260924 12:48:25.320842 11499 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f9fab5de21c4290920a69f4d73c3ef2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:48:25.320962 11499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f9fab5de21c4290920a69f4d73c3ef2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:48:25.321092 11499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f9fab5de21c4290920a69f4d73c3ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:25.323124 11499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f9fab5de21c4290920a69f4d73c3ef2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9fab5de21c4290920a69f4d73c3ef2" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 39827 } }
I20260924 12:48:25.323567 11499 leader_election.cc:302] T 00000000000000000000000000000000 P 3f9fab5de21c4290920a69f4d73c3ef2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f9fab5de21c4290920a69f4d73c3ef2; no voters:
I20260924 12:48:25.324079 11499 leader_election.cc:288] T 00000000000000000000000000000000 P 3f9fab5de21c4290920a69f4d73c3ef2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:48:25.324210 11504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f9fab5de21c4290920a69f4d73c3ef2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:25.324887 11504 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f9fab5de21c4290920a69f4d73c3ef2 [term 1 LEADER]: Becoming Leader. State: Replica: 3f9fab5de21c4290920a69f4d73c3ef2, State: Running, Role: LEADER
I20260924 12:48:25.325444 11504 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f9fab5de21c4290920a69f4d73c3ef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9fab5de21c4290920a69f4d73c3ef2" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 39827 } }
I20260924 12:48:25.325989 11499 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f9fab5de21c4290920a69f4d73c3ef2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:48:25.327914 11506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f9fab5de21c4290920a69f4d73c3ef2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f9fab5de21c4290920a69f4d73c3ef2. Latest consensus state: current_term: 1 leader_uuid: "3f9fab5de21c4290920a69f4d73c3ef2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9fab5de21c4290920a69f4d73c3ef2" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 39827 } } }
I20260924 12:48:25.328275 11506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f9fab5de21c4290920a69f4d73c3ef2 [sys.catalog]: This master's current role is: LEADER
I20260924 12:48:25.328612 11505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f9fab5de21c4290920a69f4d73c3ef2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f9fab5de21c4290920a69f4d73c3ef2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9fab5de21c4290920a69f4d73c3ef2" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 39827 } } }
I20260924 12:48:25.328899 11505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f9fab5de21c4290920a69f4d73c3ef2 [sys.catalog]: This master's current role is: LEADER
I20260924 12:48:25.330564 11509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:48:25.338078 11509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:48:25.348806 11509 catalog_manager.cc:1347] Generated new cluster ID: 8f6d7d2efb734bc1870fc6ad715a51cd
I20260924 12:48:25.348981 11509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:48:25.365113 11509 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:48:25.367251 11509 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:48:25.382587 11509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f9fab5de21c4290920a69f4d73c3ef2: Generated new TSK 0
I20260924 12:48:25.383487 11509 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:48:25.400911 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.193:0
--local_ip_for_outbound_sockets=127.5.77.193
--webserver_interface=127.5.77.193
--webserver_port=0
--tserver_master_addrs=127.5.77.254:39827
--builtin_ntp_servers=127.5.77.212:44827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--consensus_rpc_timeout_ms=15000 with env {}
W20260924 12:48:25.723433 11523 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 12:48:25.723775 11523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:25.723865 11523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:25.723964 11523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:25.731550 11523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:25.731756 11523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.193
I20260924 12:48:25.741140 11523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:44827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=15000
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.77.193
--webserver_port=0
--tserver_master_addrs=127.5.77.254:39827
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.193
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:25.742412 11523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:25.744309 11523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:25.757128 11528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:25.757398 11531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:25.757153 11529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:25.757949 11523 server_base.cc:1034] running on GCE node
I20260924 12:48:25.758701 11523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:25.759752 11523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:25.760979 11523 hybrid_clock.cc:648] HybridClock initialized: now 1790254105760930 us; error 42 us; skew 500 ppm
I20260924 12:48:25.761304 11523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:25.763574 11523 webserver.cc:460] Webserver started at http://127.5.77.193:42067/ using document root <none> and password file <none>
I20260924 12:48:25.764257 11523 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:25.764379 11523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:25.764674 11523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:25.767331 11523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "fb47c9ff31994840b6ca98ae5f5db5ab"
format_stamp: "Formatted at 2026-09-24 12:48:25 on dist-test-slave-b8q3"
I20260924 12:48:25.768098 11523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "fb47c9ff31994840b6ca98ae5f5db5ab"
format_stamp: "Formatted at 2026-09-24 12:48:25 on dist-test-slave-b8q3"
I20260924 12:48:25.773669 11523 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 12:48:25.777333 11537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:25.778843 11523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 12:48:25.779006 11523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "fb47c9ff31994840b6ca98ae5f5db5ab"
format_stamp: "Formatted at 2026-09-24 12:48:25 on dist-test-slave-b8q3"
I20260924 12:48:25.779213 11523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:25.791991 11523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:25.792809 11523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:25.793079 11523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:25.794065 11523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:25.795472 11523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:48:25.795563 11523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:25.795665 11523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:48:25.795732 11523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:25.824347 11523 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.193:41541
I20260924 12:48:25.824384 11649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.193:41541 every 8 connection(s)
I20260924 12:48:25.825580 11523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260924 12:48:25.831322 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 11523
I20260924 12:48:25.831525 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260924 12:48:25.835144 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.194:0
--local_ip_for_outbound_sockets=127.5.77.194
--webserver_interface=127.5.77.194
--webserver_port=0
--tserver_master_addrs=127.5.77.254:39827
--builtin_ntp_servers=127.5.77.212:44827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--consensus_rpc_timeout_ms=15000 with env {}
I20260924 12:48:25.840658 11650 heartbeater.cc:347] Connected to a master server at 127.5.77.254:39827
I20260924 12:48:25.840982 11650 heartbeater.cc:464] Registering TS with master...
I20260924 12:48:25.841835 11650 heartbeater.cc:511] Master 127.5.77.254:39827 requested a full tablet report, sending...
I20260924 12:48:25.844105 11464 ts_manager.cc:194] Registered new tserver with Master: fb47c9ff31994840b6ca98ae5f5db5ab (127.5.77.193:41541)
I20260924 12:48:25.845997 11464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.193:35963
W20260924 12:48:26.155875 11654 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 12:48:26.156191 11654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:26.156256 11654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:26.156327 11654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:26.163903 11654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:26.164072 11654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.194
I20260924 12:48:26.173267 11654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:44827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=15000
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.77.194
--webserver_port=0
--tserver_master_addrs=127.5.77.254:39827
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.194
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:26.174510 11654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:26.176203 11654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:26.188495 11660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:26.188532 11659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:26.188555 11662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:26.189661 11654 server_base.cc:1034] running on GCE node
I20260924 12:48:26.190290 11654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:26.191279 11654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:26.192502 11654 hybrid_clock.cc:648] HybridClock initialized: now 1790254106192444 us; error 53 us; skew 500 ppm
I20260924 12:48:26.192835 11654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:26.195071 11654 webserver.cc:460] Webserver started at http://127.5.77.194:41741/ using document root <none> and password file <none>
I20260924 12:48:26.195753 11654 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:26.195853 11654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:26.196151 11654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:26.200456 11654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "41d5f2068405488eb763bd094f82fc1f"
format_stamp: "Formatted at 2026-09-24 12:48:26 on dist-test-slave-b8q3"
I20260924 12:48:26.201287 11654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "41d5f2068405488eb763bd094f82fc1f"
format_stamp: "Formatted at 2026-09-24 12:48:26 on dist-test-slave-b8q3"
I20260924 12:48:26.207050 11654 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260924 12:48:26.210920 11668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:26.212402 11654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 12:48:26.212587 11654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "41d5f2068405488eb763bd094f82fc1f"
format_stamp: "Formatted at 2026-09-24 12:48:26 on dist-test-slave-b8q3"
I20260924 12:48:26.212810 11654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:26.230085 11654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:26.231082 11654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:26.231398 11654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:26.232409 11654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:26.233897 11654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:48:26.234010 11654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:26.234138 11654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:48:26.234215 11654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:26.265511 11654 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.194:39739
I20260924 12:48:26.265550 11780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.194:39739 every 8 connection(s)
I20260924 12:48:26.266814 11654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260924 12:48:26.269447 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 11654
I20260924 12:48:26.269582 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260924 12:48:26.272401 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.195:0
--local_ip_for_outbound_sockets=127.5.77.195
--webserver_interface=127.5.77.195
--webserver_port=0
--tserver_master_addrs=127.5.77.254:39827
--builtin_ntp_servers=127.5.77.212:44827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--consensus_rpc_timeout_ms=15000 with env {}
I20260924 12:48:26.280243 11781 heartbeater.cc:347] Connected to a master server at 127.5.77.254:39827
I20260924 12:48:26.280589 11781 heartbeater.cc:464] Registering TS with master...
I20260924 12:48:26.281447 11781 heartbeater.cc:511] Master 127.5.77.254:39827 requested a full tablet report, sending...
I20260924 12:48:26.282867 11464 ts_manager.cc:194] Registered new tserver with Master: 41d5f2068405488eb763bd094f82fc1f (127.5.77.194:39739)
I20260924 12:48:26.283675 11464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.194:54733
W20260924 12:48:26.597240 11784 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 12:48:26.597561 11784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:26.597643 11784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:26.597741 11784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:26.605062 11784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:26.605229 11784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.195
I20260924 12:48:26.614396 11784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:44827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=15000
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.77.195
--webserver_port=0
--tserver_master_addrs=127.5.77.254:39827
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.195
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:26.615828 11784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:26.617614 11784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:26.630781 11789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:26.630716 11790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:26.630751 11792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:26.631973 11784 server_base.cc:1034] running on GCE node
I20260924 12:48:26.632715 11784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:26.633699 11784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:26.634913 11784 hybrid_clock.cc:648] HybridClock initialized: now 1790254106634858 us; error 37 us; skew 500 ppm
I20260924 12:48:26.635222 11784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:26.637539 11784 webserver.cc:460] Webserver started at http://127.5.77.195:34931/ using document root <none> and password file <none>
I20260924 12:48:26.638242 11784 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:26.638348 11784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:26.638697 11784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:26.641292 11784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "beeb941434d34ae68ea47066fc270484"
format_stamp: "Formatted at 2026-09-24 12:48:26 on dist-test-slave-b8q3"
I20260924 12:48:26.642056 11784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "beeb941434d34ae68ea47066fc270484"
format_stamp: "Formatted at 2026-09-24 12:48:26 on dist-test-slave-b8q3"
I20260924 12:48:26.647446 11784 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 12:48:26.650902 11798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:26.652207 11784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 12:48:26.652354 11784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "beeb941434d34ae68ea47066fc270484"
format_stamp: "Formatted at 2026-09-24 12:48:26 on dist-test-slave-b8q3"
I20260924 12:48:26.652544 11784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:26.672533 11784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:26.673412 11784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:26.673720 11784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:26.674714 11784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:26.676186 11784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:48:26.676280 11784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:26.676386 11784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:48:26.676453 11784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:26.707015 11784 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.195:35261
I20260924 12:48:26.707070 11910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.195:35261 every 8 connection(s)
I20260924 12:48:26.708384 11784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260924 12:48:26.714634 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 11784
I20260924 12:48:26.714811 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260924 12:48:26.722426 11911 heartbeater.cc:347] Connected to a master server at 127.5.77.254:39827
I20260924 12:48:26.722769 11911 heartbeater.cc:464] Registering TS with master...
I20260924 12:48:26.723541 11911 heartbeater.cc:511] Master 127.5.77.254:39827 requested a full tablet report, sending...
I20260924 12:48:26.724941 11464 ts_manager.cc:194] Registered new tserver with Master: beeb941434d34ae68ea47066fc270484 (127.5.77.195:35261)
I20260924 12:48:26.725778 11464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.195:35093
I20260924 12:48:26.730123 5431 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:48:26.745186 11464 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51252:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20260924 12:48:26.751886 11464 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:48:26.781311 11716 tablet_service.cc:1467] Processing CreateTablet for tablet 1a6084877ed945b1b6b1546b2f42de8c (DEFAULT_TABLE table=TestTable [id=e0f92dabca934419898817c38db91c8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:48:26.782840 11716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a6084877ed945b1b6b1546b2f42de8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:26.786521 11585 tablet_service.cc:1467] Processing CreateTablet for tablet 1a6084877ed945b1b6b1546b2f42de8c (DEFAULT_TABLE table=TestTable [id=e0f92dabca934419898817c38db91c8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:48:26.787943 11585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a6084877ed945b1b6b1546b2f42de8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:26.788800 11846 tablet_service.cc:1467] Processing CreateTablet for tablet 1a6084877ed945b1b6b1546b2f42de8c (DEFAULT_TABLE table=TestTable [id=e0f92dabca934419898817c38db91c8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:48:26.790136 11846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a6084877ed945b1b6b1546b2f42de8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:26.794123 11929 tablet_bootstrap.cc:492] T 1a6084877ed945b1b6b1546b2f42de8c P 41d5f2068405488eb763bd094f82fc1f: Bootstrap starting.
I20260924 12:48:26.797806 11929 tablet_bootstrap.cc:654] T 1a6084877ed945b1b6b1546b2f42de8c P 41d5f2068405488eb763bd094f82fc1f: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:26.798853 11929 log.cc:824] T 1a6084877ed945b1b6b1546b2f42de8c P 41d5f2068405488eb763bd094f82fc1f: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:26.801298 11929 tablet_bootstrap.cc:492] T 1a6084877ed945b1b6b1546b2f42de8c P 41d5f2068405488eb763bd094f82fc1f: No bootstrap required, opened a new log
I20260924 12:48:26.801666 11929 ts_tablet_manager.cc:1397] T 1a6084877ed945b1b6b1546b2f42de8c P 41d5f2068405488eb763bd094f82fc1f: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 12:48:26.802922 11931 tablet_bootstrap.cc:492] T 1a6084877ed945b1b6b1546b2f42de8c P fb47c9ff31994840b6ca98ae5f5db5ab: Bootstrap starting.
I20260924 12:48:26.803803 11932 tablet_bootstrap.cc:492] T 1a6084877ed945b1b6b1546b2f42de8c P beeb941434d34ae68ea47066fc270484: Bootstrap starting.
I20260924 12:48:26.806372 11931 tablet_bootstrap.cc:654] T 1a6084877ed945b1b6b1546b2f42de8c P fb47c9ff31994840b6ca98ae5f5db5ab: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:26.806437 11932 tablet_bootstrap.cc:654] T 1a6084877ed945b1b6b1546b2f42de8c P beeb941434d34ae68ea47066fc270484: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:26.807391 11932 log.cc:824] T 1a6084877ed945b1b6b1546b2f42de8c P beeb941434d34ae68ea47066fc270484: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:26.807585 11931 log.cc:824] T 1a6084877ed945b1b6b1546b2f42de8c P fb47c9ff31994840b6ca98ae5f5db5ab: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:26.810289 11931 tablet_bootstrap.cc:492] T 1a6084877ed945b1b6b1546b2f42de8c P fb47c9ff31994840b6ca98ae5f5db5ab: No bootstrap required, opened a new log
I20260924 12:48:26.810328 11932 tablet_bootstrap.cc:492] T 1a6084877ed945b1b6b1546b2f42de8c P beeb941434d34ae68ea47066fc270484: No bootstrap required, opened a new log
I20260924 12:48:26.810590 11931 ts_tablet_manager.cc:1397] T 1a6084877ed945b1b6b1546b2f42de8c P fb47c9ff31994840b6ca98ae5f5db5ab: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260924 12:48:26.810616 11932 ts_tablet_manager.cc:1397] T 1a6084877ed945b1b6b1546b2f42de8c P beeb941434d34ae68ea47066fc270484: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260924 12:48:26.811590 11929 raft_consensus.cc:348] T 1a6084877ed945b1b6b1546b2f42de8c P 41d5f2068405488eb763bd094f82fc1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d5f2068405488eb763bd094f82fc1f" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39739 } } peers { permanent_uuid: "beeb941434d34ae68ea47066fc270484" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35261 } } peers { permanent_uuid: "fb47c9ff31994840b6ca98ae5f5db5ab" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41541 } }
I20260924 12:48:26.812074 11929 raft_consensus.cc:729] T 1a6084877ed945b1b6b1546b2f42de8c P 41d5f2068405488eb763bd094f82fc1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41d5f2068405488eb763bd094f82fc1f, State: Initialized, Role: FOLLOWER
I20260924 12:48:26.812544 11929 consensus_queue.cc:260] T 1a6084877ed945b1b6b1546b2f42de8c P 41d5f2068405488eb763bd094f82fc1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d5f2068405488eb763bd094f82fc1f" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39739 } } peers { permanent_uuid: "beeb941434d34ae68ea47066fc270484" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35261 } } peers { permanent_uuid: "fb47c9ff31994840b6ca98ae5f5db5ab" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41541 } }
I20260924 12:48:26.813699 11781 heartbeater.cc:503] Master 127.5.77.254:39827 was elected leader, sending a full tablet report...
I20260924 12:48:26.814354 11929 ts_tablet_manager.cc:1428] T 1a6084877ed945b1b6b1546b2f42de8c P 41d5f2068405488eb763bd094f82fc1f: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260924 12:48:26.818053 11931 raft_consensus.cc:348] T 1a6084877ed945b1b6b1546b2f42de8c P fb47c9ff31994840b6ca98ae5f5db5ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d5f2068405488eb763bd094f82fc1f" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39739 } } peers { permanent_uuid: "beeb941434d34ae68ea47066fc270484" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35261 } } peers { permanent_uuid: "fb47c9ff31994840b6ca98ae5f5db5ab" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41541 } }
I20260924 12:48:26.818565 11931 raft_consensus.cc:729] T 1a6084877ed945b1b6b1546b2f42de8c P fb47c9ff31994840b6ca98ae5f5db5ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb47c9ff31994840b6ca98ae5f5db5ab, State: Initialized, Role: FOLLOWER
I20260924 12:48:26.819058 11931 consensus_queue.cc:260] T 1a6084877ed945b1b6b1546b2f42de8c P fb47c9ff31994840b6ca98ae5f5db5ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d5f2068405488eb763bd094f82fc1f" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39739 } } peers { permanent_uuid: "beeb941434d34ae68ea47066fc270484" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35261 } } peers { permanent_uuid: "fb47c9ff31994840b6ca98ae5f5db5ab" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41541 } }
I20260924 12:48:26.820765 11931 ts_tablet_manager.cc:1428] T 1a6084877ed945b1b6b1546b2f42de8c P fb47c9ff31994840b6ca98ae5f5db5ab: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260924 12:48:26.821074 11650 heartbeater.cc:503] Master 127.5.77.254:39827 was elected leader, sending a full tablet report...
I20260924 12:48:26.821533 11932 raft_consensus.cc:348] T 1a6084877ed945b1b6b1546b2f42de8c P beeb941434d34ae68ea47066fc270484 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d5f2068405488eb763bd094f82fc1f" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39739 } } peers { permanent_uuid: "beeb941434d34ae68ea47066fc270484" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35261 } } peers { permanent_uuid: "fb47c9ff31994840b6ca98ae5f5db5ab" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41541 } }
I20260924 12:48:26.821997 11932 raft_consensus.cc:729] T 1a6084877ed945b1b6b1546b2f42de8c P beeb941434d34ae68ea47066fc270484 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beeb941434d34ae68ea47066fc270484, State: Initialized, Role: FOLLOWER
I20260924 12:48:26.822438 11932 consensus_queue.cc:260] T 1a6084877ed945b1b6b1546b2f42de8c P beeb941434d34ae68ea47066fc270484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d5f2068405488eb763bd094f82fc1f" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39739 } } peers { permanent_uuid: "beeb941434d34ae68ea47066fc270484" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35261 } } peers { permanent_uuid: "fb47c9ff31994840b6ca98ae5f5db5ab" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41541 } }
I20260924 12:48:26.824321 11932 ts_tablet_manager.cc:1428] T 1a6084877ed945b1b6b1546b2f42de8c P beeb941434d34ae68ea47066fc270484: Time spent starting tablet: real 0.014s user 0.006s sys 0.005s
I20260924 12:48:26.824478 11911 heartbeater.cc:503] Master 127.5.77.254:39827 was elected leader, sending a full tablet report...
W20260924 12:48:26.827728 11651 tablet.cc:2369] T 1a6084877ed945b1b6b1546b2f42de8c P fb47c9ff31994840b6ca98ae5f5db5ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:48:26.834491 5431 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:48:26.836447 5431 ts_itest-base.cc:233] Waiting for 1 tablets on tserver fb47c9ff31994840b6ca98ae5f5db5ab to finish bootstrapping
I20260924 12:48:26.845149 5431 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 41d5f2068405488eb763bd094f82fc1f to finish bootstrapping
I20260924 12:48:26.852921 5431 ts_itest-base.cc:233] Waiting for 1 tablets on tserver beeb941434d34ae68ea47066fc270484 to finish bootstrapping
I20260924 12:48:26.866823 11736 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1a6084877ed945b1b6b1546b2f42de8c"
dest_uuid: "41d5f2068405488eb763bd094f82fc1f"
from {username='slave'} at 127.0.0.1:36598
I20260924 12:48:26.867146 11736 raft_consensus.cc:482] T 1a6084877ed945b1b6b1546b2f42de8c P 41d5f2068405488eb763bd094f82fc1f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 12:48:26.867322 11736 raft_consensus.cc:3037] T 1a6084877ed945b1b6b1546b2f42de8c P 41d5f2068405488eb763bd094f82fc1f [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:26.869390 11736 raft_consensus.cc:504] T 1a6084877ed945b1b6b1546b2f42de8c P 41d5f2068405488eb763bd094f82fc1f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d5f2068405488eb763bd094f82fc1f" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39739 } } peers { permanent_uuid: "beeb941434d34ae68ea47066fc270484" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35261 } } peers { permanent_uuid: "fb47c9ff31994840b6ca98ae5f5db5ab" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41541 } }
I20260924 12:48:26.871158 11736 leader_election.cc:288] T 1a6084877ed945b1b6b1546b2f42de8c P 41d5f2068405488eb763bd094f82fc1f [CANDIDATE]: Term 1 election: Requested vote from peers beeb941434d34ae68ea47066fc270484 (127.5.77.195:35261), fb47c9ff31994840b6ca98ae5f5db5ab (127.5.77.193:41541)
I20260924 12:48:26.882314 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a6084877ed945b1b6b1546b2f42de8c" candidate_uuid: "41d5f2068405488eb763bd094f82fc1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "beeb941434d34ae68ea47066fc270484"
I20260924 12:48:26.882350 11605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a6084877ed945b1b6b1546b2f42de8c" candidate_uuid: "41d5f2068405488eb763bd094f82fc1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fb47c9ff31994840b6ca98ae5f5db5ab"
I20260924 12:48:26.882745 11605 raft_consensus.cc:3037] T 1a6084877ed945b1b6b1546b2f42de8c P fb47c9ff31994840b6ca98ae5f5db5ab [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:26.882747 11866 raft_consensus.cc:3037] T 1a6084877ed945b1b6b1546b2f42de8c P beeb941434d34ae68ea47066fc270484 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:26.884886 11866 raft_consensus.cc:2454] T 1a6084877ed945b1b6b1546b2f42de8c P beeb941434d34ae68ea47066fc270484 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41d5f2068405488eb763bd094f82fc1f in term 1.
I20260924 12:48:26.885025 11605 raft_consensus.cc:2454] T 1a6084877ed945b1b6b1546b2f42de8c P fb47c9ff31994840b6ca98ae5f5db5ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41d5f2068405488eb763bd094f82fc1f in term 1.
I20260924 12:48:26.885548 11672 leader_election.cc:302] T 1a6084877ed945b1b6b1546b2f42de8c P 41d5f2068405488eb763bd094f82fc1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41d5f2068405488eb763bd094f82fc1f, beeb941434d34ae68ea47066fc270484; no voters:
I20260924 12:48:26.885965 11935 raft_consensus.cc:2781] T 1a6084877ed945b1b6b1546b2f42de8c P 41d5f2068405488eb763bd094f82fc1f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:26.886618 11935 raft_consensus.cc:686] T 1a6084877ed945b1b6b1546b2f42de8c P 41d5f2068405488eb763bd094f82fc1f [term 1 LEADER]: Becoming Leader. State: Replica: 41d5f2068405488eb763bd094f82fc1f, State: Running, Role: LEADER
I20260924 12:48:26.887168 11935 consensus_queue.cc:237] T 1a6084877ed945b1b6b1546b2f42de8c P 41d5f2068405488eb763bd094f82fc1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d5f2068405488eb763bd094f82fc1f" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39739 } } peers { permanent_uuid: "beeb941434d34ae68ea47066fc270484" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35261 } } peers { permanent_uuid: "fb47c9ff31994840b6ca98ae5f5db5ab" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41541 } }
I20260924 12:48:26.892287 11464 catalog_manager.cc:5495] T 1a6084877ed945b1b6b1546b2f42de8c P 41d5f2068405488eb763bd094f82fc1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 41d5f2068405488eb763bd094f82fc1f (127.5.77.194). New cstate: current_term: 1 leader_uuid: "41d5f2068405488eb763bd094f82fc1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d5f2068405488eb763bd094f82fc1f" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39739 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "beeb941434d34ae68ea47066fc270484" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb47c9ff31994840b6ca98ae5f5db5ab" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41541 } health_report { overall_health: UNKNOWN } } }
W20260924 12:48:27.018666 11782 tablet.cc:2369] T 1a6084877ed945b1b6b1546b2f42de8c P 41d5f2068405488eb763bd094f82fc1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:48:27.913939 11736 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1a6084877ed945b1b6b1546b2f42de8c"
dest_uuid: "41d5f2068405488eb763bd094f82fc1f"
from {username='slave'} at 127.0.0.1:36598
I20260924 12:48:27.914234 11736 raft_consensus.cc:580] T 1a6084877ed945b1b6b1546b2f42de8c P 41d5f2068405488eb763bd094f82fc1f [term 1 LEADER]: Received request to step down
I20260924 12:48:27.914317 11736 raft_consensus.cc:3032] T 1a6084877ed945b1b6b1546b2f42de8c P 41d5f2068405488eb763bd094f82fc1f [term 1 LEADER]: Stepping down as leader of term 1
I20260924 12:48:27.914390 11736 raft_consensus.cc:729] T 1a6084877ed945b1b6b1546b2f42de8c P 41d5f2068405488eb763bd094f82fc1f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 41d5f2068405488eb763bd094f82fc1f, State: Running, Role: LEADER
I20260924 12:48:27.914634 11736 consensus_queue.cc:260] T 1a6084877ed945b1b6b1546b2f42de8c P 41d5f2068405488eb763bd094f82fc1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d5f2068405488eb763bd094f82fc1f" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39739 } } peers { permanent_uuid: "beeb941434d34ae68ea47066fc270484" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35261 } } peers { permanent_uuid: "fb47c9ff31994840b6ca98ae5f5db5ab" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 41541 } }
I20260924 12:48:27.914892 11736 raft_consensus.cc:3037] T 1a6084877ed945b1b6b1546b2f42de8c P 41d5f2068405488eb763bd094f82fc1f [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:48:27.916230 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 11523
W20260924 12:48:27.928614 11670 connection.cc:535] client connection to 127.5.77.193:41541 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 12:48:27.928910 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 11654
W20260924 12:48:27.929019 11670 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 12:48:27.930068 11776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:28.020020 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 11654
I20260924 12:48:28.033761 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 11784
I20260924 12:48:28.045759 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 11433
I20260924 12:48:28.046995 11494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:28.149924 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 11433
2026-09-24T12:48:28Z chronyd exiting
[ OK ] RaftConsensusElectionITest.StepDownWithSlowFollower (3356 ms)
[ RUN ] RaftConsensusElectionITest.ElectPendingVoter
I20260924 12:48:28.185420 5431 test_util.cc:268] Using random seed: 70062881
I20260924 12:48:28.187148 5431 ts_itest-base.cc:112] Starting cluster with:
I20260924 12:48:28.187229 5431 ts_itest-base.cc:113] --------------
I20260924 12:48:28.187278 5431 ts_itest-base.cc:114] 5 tablet servers
I20260924 12:48:28.187336 5431 ts_itest-base.cc:115] 5 replicas per TS
I20260924 12:48:28.187383 5431 ts_itest-base.cc:116] --------------
2026-09-24T12:48:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:48:28Z Disabled control of system clock
I20260924 12:48:28.217200 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.77.254:33639
--webserver_interface=127.5.77.254
--webserver_port=0
--builtin_ntp_servers=127.5.77.212:36627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.77.254:33639
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20260924 12:48:28.538524 11955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:28.538913 11955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:28.539031 11955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:28.546432 11955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:48:28.546572 11955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:28.546623 11955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:48:28.546694 11955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:48:28.556052 11955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:36627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.5.77.254:33639
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.77.254:33639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.77.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:28.557438 11955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:28.559274 11955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:28.570737 11960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:28.570698 11961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:28.570815 11963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:28.571278 11955 server_base.cc:1034] running on GCE node
I20260924 12:48:28.572029 11955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:28.573518 11955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:28.574734 11955 hybrid_clock.cc:648] HybridClock initialized: now 1790254108574668 us; error 44 us; skew 500 ppm
I20260924 12:48:28.575045 11955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:28.577325 11955 webserver.cc:460] Webserver started at http://127.5.77.254:45325/ using document root <none> and password file <none>
I20260924 12:48:28.577960 11955 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:28.578070 11955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:28.578351 11955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:28.580919 11955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "69010a3867ee49ebb8a7f5342ad4b536"
format_stamp: "Formatted at 2026-09-24 12:48:28 on dist-test-slave-b8q3"
I20260924 12:48:28.581611 11955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "69010a3867ee49ebb8a7f5342ad4b536"
format_stamp: "Formatted at 2026-09-24 12:48:28 on dist-test-slave-b8q3"
I20260924 12:48:28.586777 11955 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 12:48:28.591142 11969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:28.592844 11955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:48:28.592994 11955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
uuid: "69010a3867ee49ebb8a7f5342ad4b536"
format_stamp: "Formatted at 2026-09-24 12:48:28 on dist-test-slave-b8q3"
I20260924 12:48:28.593199 11955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:28.612872 11955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:28.613739 11955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:28.614017 11955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:28.631215 11955 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.254:33639
I20260924 12:48:28.631237 12020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.254:33639 every 8 connection(s)
I20260924 12:48:28.632624 11955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260924 12:48:28.636521 12021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:28.637913 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 11955
I20260924 12:48:28.638116 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260924 12:48:28.647446 12021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69010a3867ee49ebb8a7f5342ad4b536: Bootstrap starting.
I20260924 12:48:28.651016 12021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69010a3867ee49ebb8a7f5342ad4b536: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:28.652093 12021 log.cc:824] T 00000000000000000000000000000000 P 69010a3867ee49ebb8a7f5342ad4b536: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:28.654773 12021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69010a3867ee49ebb8a7f5342ad4b536: No bootstrap required, opened a new log
I20260924 12:48:28.662101 12021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69010a3867ee49ebb8a7f5342ad4b536 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69010a3867ee49ebb8a7f5342ad4b536" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 33639 } }
I20260924 12:48:28.662566 12021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69010a3867ee49ebb8a7f5342ad4b536 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:48:28.662657 12021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69010a3867ee49ebb8a7f5342ad4b536 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69010a3867ee49ebb8a7f5342ad4b536, State: Initialized, Role: FOLLOWER
I20260924 12:48:28.663168 12021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69010a3867ee49ebb8a7f5342ad4b536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69010a3867ee49ebb8a7f5342ad4b536" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 33639 } }
I20260924 12:48:28.663357 12021 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69010a3867ee49ebb8a7f5342ad4b536 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:48:28.663452 12021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69010a3867ee49ebb8a7f5342ad4b536 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:48:28.663568 12021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69010a3867ee49ebb8a7f5342ad4b536 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:28.665493 12021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69010a3867ee49ebb8a7f5342ad4b536 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69010a3867ee49ebb8a7f5342ad4b536" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 33639 } }
I20260924 12:48:28.665957 12021 leader_election.cc:302] T 00000000000000000000000000000000 P 69010a3867ee49ebb8a7f5342ad4b536 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69010a3867ee49ebb8a7f5342ad4b536; no voters:
I20260924 12:48:28.666507 12021 leader_election.cc:288] T 00000000000000000000000000000000 P 69010a3867ee49ebb8a7f5342ad4b536 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:48:28.666668 12026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69010a3867ee49ebb8a7f5342ad4b536 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:28.667757 12026 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69010a3867ee49ebb8a7f5342ad4b536 [term 1 LEADER]: Becoming Leader. State: Replica: 69010a3867ee49ebb8a7f5342ad4b536, State: Running, Role: LEADER
I20260924 12:48:28.668318 12026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69010a3867ee49ebb8a7f5342ad4b536 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69010a3867ee49ebb8a7f5342ad4b536" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 33639 } }
I20260924 12:48:28.668815 12021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69010a3867ee49ebb8a7f5342ad4b536 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:48:28.671078 12028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69010a3867ee49ebb8a7f5342ad4b536 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69010a3867ee49ebb8a7f5342ad4b536. Latest consensus state: current_term: 1 leader_uuid: "69010a3867ee49ebb8a7f5342ad4b536" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69010a3867ee49ebb8a7f5342ad4b536" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 33639 } } }
I20260924 12:48:28.671382 12027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69010a3867ee49ebb8a7f5342ad4b536 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69010a3867ee49ebb8a7f5342ad4b536" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69010a3867ee49ebb8a7f5342ad4b536" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 33639 } } }
I20260924 12:48:28.671660 12028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69010a3867ee49ebb8a7f5342ad4b536 [sys.catalog]: This master's current role is: LEADER
I20260924 12:48:28.671725 12027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69010a3867ee49ebb8a7f5342ad4b536 [sys.catalog]: This master's current role is: LEADER
I20260924 12:48:28.676209 12036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:48:28.681612 12036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:48:28.690140 12036 catalog_manager.cc:1347] Generated new cluster ID: 686bc782635b42649619a54f5d51428c
I20260924 12:48:28.690281 12036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:48:28.711129 12036 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:48:28.712509 12036 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:48:28.720068 12036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69010a3867ee49ebb8a7f5342ad4b536: Generated new TSK 0
I20260924 12:48:28.720973 12036 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:48:28.735720 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.193:0
--local_ip_for_outbound_sockets=127.5.77.193
--webserver_interface=127.5.77.193
--webserver_port=0
--tserver_master_addrs=127.5.77.254:33639
--builtin_ntp_servers=127.5.77.212:36627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20260924 12:48:29.067763 12045 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 12:48:29.068122 12045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:29.068212 12045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:29.068313 12045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:29.076007 12045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:29.076202 12045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.193
I20260924 12:48:29.085603 12045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:36627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.77.193
--webserver_port=0
--tserver_master_addrs=127.5.77.254:33639
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.193
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:29.087033 12045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:29.088838 12045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:29.103011 12053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:29.103114 12050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:29.103027 12051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:29.103837 12045 server_base.cc:1034] running on GCE node
I20260924 12:48:29.104442 12045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:29.105382 12045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:29.106556 12045 hybrid_clock.cc:648] HybridClock initialized: now 1790254109106491 us; error 55 us; skew 500 ppm
I20260924 12:48:29.106886 12045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:29.116151 12045 webserver.cc:460] Webserver started at http://127.5.77.193:44721/ using document root <none> and password file <none>
I20260924 12:48:29.116986 12045 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:29.117120 12045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:29.117548 12045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:29.120643 12045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "0cff88768c07440e8aa9fe05f868c706"
format_stamp: "Formatted at 2026-09-24 12:48:29 on dist-test-slave-b8q3"
I20260924 12:48:29.121522 12045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "0cff88768c07440e8aa9fe05f868c706"
format_stamp: "Formatted at 2026-09-24 12:48:29 on dist-test-slave-b8q3"
I20260924 12:48:29.127432 12045 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 12:48:29.131167 12059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:29.132673 12045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:48:29.132889 12045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "0cff88768c07440e8aa9fe05f868c706"
format_stamp: "Formatted at 2026-09-24 12:48:29 on dist-test-slave-b8q3"
I20260924 12:48:29.133078 12045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:29.148699 12045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:29.149569 12045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:29.149842 12045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:29.150792 12045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:29.152315 12045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:48:29.152402 12045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:29.152487 12045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:48:29.152539 12045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:29.182775 12045 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.193:40467
I20260924 12:48:29.182799 12171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.193:40467 every 8 connection(s)
I20260924 12:48:29.184069 12045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260924 12:48:29.189038 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 12045
I20260924 12:48:29.189175 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260924 12:48:29.192147 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.194:0
--local_ip_for_outbound_sockets=127.5.77.194
--webserver_interface=127.5.77.194
--webserver_port=0
--tserver_master_addrs=127.5.77.254:33639
--builtin_ntp_servers=127.5.77.212:36627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260924 12:48:29.197764 12172 heartbeater.cc:347] Connected to a master server at 127.5.77.254:33639
I20260924 12:48:29.198076 12172 heartbeater.cc:464] Registering TS with master...
I20260924 12:48:29.198880 12172 heartbeater.cc:511] Master 127.5.77.254:33639 requested a full tablet report, sending...
I20260924 12:48:29.201122 11986 ts_manager.cc:194] Registered new tserver with Master: 0cff88768c07440e8aa9fe05f868c706 (127.5.77.193:40467)
I20260924 12:48:29.203091 11986 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.193:48787
W20260924 12:48:29.517295 12176 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 12:48:29.517611 12176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:29.517674 12176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:29.517752 12176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:29.525152 12176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:29.525311 12176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.194
I20260924 12:48:29.534377 12176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:36627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.77.194
--webserver_port=0
--tserver_master_addrs=127.5.77.254:33639
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.194
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:29.536000 12176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:29.537837 12176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:29.551071 12181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:29.551108 12182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:29.551136 12184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:29.555811 12176 server_base.cc:1034] running on GCE node
I20260924 12:48:29.556509 12176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:29.557542 12176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:29.558756 12176 hybrid_clock.cc:648] HybridClock initialized: now 1790254109558695 us; error 62 us; skew 500 ppm
I20260924 12:48:29.559070 12176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:29.561357 12176 webserver.cc:460] Webserver started at http://127.5.77.194:43329/ using document root <none> and password file <none>
I20260924 12:48:29.562047 12176 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:29.562150 12176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:29.562477 12176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:29.565145 12176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "4a09a2719ee244818f474ff51937f75d"
format_stamp: "Formatted at 2026-09-24 12:48:29 on dist-test-slave-b8q3"
I20260924 12:48:29.565917 12176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "4a09a2719ee244818f474ff51937f75d"
format_stamp: "Formatted at 2026-09-24 12:48:29 on dist-test-slave-b8q3"
I20260924 12:48:29.571386 12176 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260924 12:48:29.574821 12190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:29.576171 12176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 12:48:29.576347 12176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "4a09a2719ee244818f474ff51937f75d"
format_stamp: "Formatted at 2026-09-24 12:48:29 on dist-test-slave-b8q3"
I20260924 12:48:29.576543 12176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:29.591940 12176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:29.592797 12176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:29.593080 12176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:29.594019 12176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:29.595472 12176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:48:29.595566 12176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:29.595672 12176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:48:29.595739 12176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:29.625078 12176 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.194:46345
I20260924 12:48:29.625128 12302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.194:46345 every 8 connection(s)
I20260924 12:48:29.626425 12176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260924 12:48:29.635882 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 12176
I20260924 12:48:29.636080 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260924 12:48:29.639420 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.195:0
--local_ip_for_outbound_sockets=127.5.77.195
--webserver_interface=127.5.77.195
--webserver_port=0
--tserver_master_addrs=127.5.77.254:33639
--builtin_ntp_servers=127.5.77.212:36627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260924 12:48:29.640820 12303 heartbeater.cc:347] Connected to a master server at 127.5.77.254:33639
I20260924 12:48:29.641129 12303 heartbeater.cc:464] Registering TS with master...
I20260924 12:48:29.641902 12303 heartbeater.cc:511] Master 127.5.77.254:33639 requested a full tablet report, sending...
I20260924 12:48:29.643254 11986 ts_manager.cc:194] Registered new tserver with Master: 4a09a2719ee244818f474ff51937f75d (127.5.77.194:46345)
I20260924 12:48:29.644081 11986 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.194:33781
W20260924 12:48:29.961287 12306 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 12:48:29.961630 12306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:29.961700 12306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:29.961771 12306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:29.969592 12306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:29.969798 12306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.195
I20260924 12:48:29.979200 12306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:36627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.77.195
--webserver_port=0
--tserver_master_addrs=127.5.77.254:33639
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.195
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:29.980600 12306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:29.982372 12306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:29.995910 12311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:29.995857 12312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:29.995829 12314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:29.996409 12306 server_base.cc:1034] running on GCE node
I20260924 12:48:29.997115 12306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:29.998109 12306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:29.999339 12306 hybrid_clock.cc:648] HybridClock initialized: now 1790254109999282 us; error 54 us; skew 500 ppm
I20260924 12:48:29.999668 12306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:30.002008 12306 webserver.cc:460] Webserver started at http://127.5.77.195:43247/ using document root <none> and password file <none>
I20260924 12:48:30.002800 12306 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:30.002905 12306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:30.003203 12306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:30.005901 12306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a72ae1f099324d0eb8b0104438256f28"
format_stamp: "Formatted at 2026-09-24 12:48:29 on dist-test-slave-b8q3"
I20260924 12:48:30.006729 12306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a72ae1f099324d0eb8b0104438256f28"
format_stamp: "Formatted at 2026-09-24 12:48:29 on dist-test-slave-b8q3"
I20260924 12:48:30.012560 12306 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 12:48:30.016736 12320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:30.018337 12306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 12:48:30.018558 12306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a72ae1f099324d0eb8b0104438256f28"
format_stamp: "Formatted at 2026-09-24 12:48:29 on dist-test-slave-b8q3"
I20260924 12:48:30.018778 12306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:30.039510 12306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:30.040393 12306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:30.040710 12306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:30.041741 12306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:30.043264 12306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:48:30.043368 12306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:30.043474 12306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:48:30.043541 12306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:30.074642 12306 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.195:37669
I20260924 12:48:30.074694 12432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.195:37669 every 8 connection(s)
I20260924 12:48:30.076066 12306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260924 12:48:30.082386 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 12306
I20260924 12:48:30.082581 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260924 12:48:30.086074 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.196:0
--local_ip_for_outbound_sockets=127.5.77.196
--webserver_interface=127.5.77.196
--webserver_port=0
--tserver_master_addrs=127.5.77.254:33639
--builtin_ntp_servers=127.5.77.212:36627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260924 12:48:30.091647 12433 heartbeater.cc:347] Connected to a master server at 127.5.77.254:33639
I20260924 12:48:30.092017 12433 heartbeater.cc:464] Registering TS with master...
I20260924 12:48:30.092995 12433 heartbeater.cc:511] Master 127.5.77.254:33639 requested a full tablet report, sending...
I20260924 12:48:30.094614 11986 ts_manager.cc:194] Registered new tserver with Master: a72ae1f099324d0eb8b0104438256f28 (127.5.77.195:37669)
I20260924 12:48:30.095767 11986 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.195:55737
I20260924 12:48:30.206344 12172 heartbeater.cc:503] Master 127.5.77.254:33639 was elected leader, sending a full tablet report...
W20260924 12:48:30.414611 12436 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 12:48:30.414937 12436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:30.415004 12436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:30.415096 12436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:30.422912 12436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:30.423090 12436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.196
I20260924 12:48:30.432507 12436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:36627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.5.77.196
--webserver_port=0
--tserver_master_addrs=127.5.77.254:33639
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.196
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:30.433825 12436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:30.435640 12436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:30.448975 12441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:30.449007 12442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:30.449051 12444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:30.449905 12436 server_base.cc:1034] running on GCE node
I20260924 12:48:30.450610 12436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:30.451615 12436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:30.452831 12436 hybrid_clock.cc:648] HybridClock initialized: now 1790254110452772 us; error 56 us; skew 500 ppm
I20260924 12:48:30.453157 12436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:30.455509 12436 webserver.cc:460] Webserver started at http://127.5.77.196:38181/ using document root <none> and password file <none>
I20260924 12:48:30.456213 12436 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:30.456314 12436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:30.456604 12436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:30.459403 12436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "8c6f70afcab64a2a946d2aad7ceda99b"
format_stamp: "Formatted at 2026-09-24 12:48:30 on dist-test-slave-b8q3"
I20260924 12:48:30.460201 12436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "8c6f70afcab64a2a946d2aad7ceda99b"
format_stamp: "Formatted at 2026-09-24 12:48:30 on dist-test-slave-b8q3"
I20260924 12:48:30.465607 12436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260924 12:48:30.469321 12450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:30.471026 12436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:48:30.471211 12436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "8c6f70afcab64a2a946d2aad7ceda99b"
format_stamp: "Formatted at 2026-09-24 12:48:30 on dist-test-slave-b8q3"
I20260924 12:48:30.471424 12436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:30.492823 12436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:30.493710 12436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:30.494001 12436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:30.494995 12436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:30.496585 12436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:48:30.496676 12436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:30.496780 12436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:48:30.496841 12436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:30.526574 12436 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.196:39661
I20260924 12:48:30.526619 12562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.196:39661 every 8 connection(s)
I20260924 12:48:30.527877 12436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20260924 12:48:30.529263 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 12436
I20260924 12:48:30.529408 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20260924 12:48:30.532591 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.197:0
--local_ip_for_outbound_sockets=127.5.77.197
--webserver_interface=127.5.77.197
--webserver_port=0
--tserver_master_addrs=127.5.77.254:33639
--builtin_ntp_servers=127.5.77.212:36627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260924 12:48:30.541839 12563 heartbeater.cc:347] Connected to a master server at 127.5.77.254:33639
I20260924 12:48:30.542192 12563 heartbeater.cc:464] Registering TS with master...
I20260924 12:48:30.543159 12563 heartbeater.cc:511] Master 127.5.77.254:33639 requested a full tablet report, sending...
I20260924 12:48:30.544936 11986 ts_manager.cc:194] Registered new tserver with Master: 8c6f70afcab64a2a946d2aad7ceda99b (127.5.77.196:39661)
I20260924 12:48:30.546037 11986 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.196:39625
I20260924 12:48:30.646631 12303 heartbeater.cc:503] Master 127.5.77.254:33639 was elected leader, sending a full tablet report...
W20260924 12:48:30.860978 12566 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 12:48:30.861343 12566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:30.861403 12566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:30.861474 12566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:30.868898 12566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:30.869066 12566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.197
I20260924 12:48:30.881198 12566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:36627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.5.77.197
--webserver_port=0
--tserver_master_addrs=127.5.77.254:33639
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.197
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:30.882494 12566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:30.884224 12566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:30.897073 12571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:30.897109 12572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:30.896984 12574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:30.898025 12566 server_base.cc:1034] running on GCE node
I20260924 12:48:30.898734 12566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:30.899740 12566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:30.900962 12566 hybrid_clock.cc:648] HybridClock initialized: now 1790254110900893 us; error 63 us; skew 500 ppm
I20260924 12:48:30.901300 12566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:30.903577 12566 webserver.cc:460] Webserver started at http://127.5.77.197:35329/ using document root <none> and password file <none>
I20260924 12:48:30.904270 12566 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:30.904374 12566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:30.904678 12566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:30.907310 12566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "4b7423e914ac4794ab465c2671f93bff"
format_stamp: "Formatted at 2026-09-24 12:48:30 on dist-test-slave-b8q3"
I20260924 12:48:30.908031 12566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "4b7423e914ac4794ab465c2671f93bff"
format_stamp: "Formatted at 2026-09-24 12:48:30 on dist-test-slave-b8q3"
I20260924 12:48:30.913326 12566 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 12:48:30.916833 12580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:30.918226 12566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 12:48:30.918375 12566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "4b7423e914ac4794ab465c2671f93bff"
format_stamp: "Formatted at 2026-09-24 12:48:30 on dist-test-slave-b8q3"
I20260924 12:48:30.918612 12566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:30.933108 12566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:30.934026 12566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:30.934327 12566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:30.935365 12566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:30.936920 12566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:48:30.937004 12566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:30.937107 12566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:48:30.937179 12566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:30.966914 12566 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.197:33815
I20260924 12:48:30.966962 12692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.197:33815 every 8 connection(s)
I20260924 12:48:30.968214 12566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20260924 12:48:30.974543 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 12566
I20260924 12:48:30.974774 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20260924 12:48:30.983803 12693 heartbeater.cc:347] Connected to a master server at 127.5.77.254:33639
I20260924 12:48:30.984167 12693 heartbeater.cc:464] Registering TS with master...
I20260924 12:48:30.985123 12693 heartbeater.cc:511] Master 127.5.77.254:33639 requested a full tablet report, sending...
I20260924 12:48:30.986831 11986 ts_manager.cc:194] Registered new tserver with Master: 4b7423e914ac4794ab465c2671f93bff (127.5.77.197:33815)
I20260924 12:48:30.987682 11986 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.197:49585
I20260924 12:48:30.990996 5431 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20260924 12:48:31.007155 11986 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36202:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20260924 12:48:31.014087 11986 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 6 tablet servers would be needed, 5 are available. Consider bringing up more tablet servers.
I20260924 12:48:31.074419 12628 tablet_service.cc:1467] Processing CreateTablet for tablet fa53dd1fa3004bd7b35e2e8fc61c0c03 (DEFAULT_TABLE table=TestTable [id=ba1835ef48dc46a38aa5a824e4889d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:48:31.074415 12238 tablet_service.cc:1467] Processing CreateTablet for tablet fa53dd1fa3004bd7b35e2e8fc61c0c03 (DEFAULT_TABLE table=TestTable [id=ba1835ef48dc46a38aa5a824e4889d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:48:31.076913 12498 tablet_service.cc:1467] Processing CreateTablet for tablet fa53dd1fa3004bd7b35e2e8fc61c0c03 (DEFAULT_TABLE table=TestTable [id=ba1835ef48dc46a38aa5a824e4889d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:48:31.077843 12368 tablet_service.cc:1467] Processing CreateTablet for tablet fa53dd1fa3004bd7b35e2e8fc61c0c03 (DEFAULT_TABLE table=TestTable [id=ba1835ef48dc46a38aa5a824e4889d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:48:31.078606 12498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa53dd1fa3004bd7b35e2e8fc61c0c03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:31.079211 12368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa53dd1fa3004bd7b35e2e8fc61c0c03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:31.080351 12238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa53dd1fa3004bd7b35e2e8fc61c0c03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:31.081203 12628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa53dd1fa3004bd7b35e2e8fc61c0c03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:31.084947 12107 tablet_service.cc:1467] Processing CreateTablet for tablet fa53dd1fa3004bd7b35e2e8fc61c0c03 (DEFAULT_TABLE table=TestTable [id=ba1835ef48dc46a38aa5a824e4889d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:48:31.086740 12107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa53dd1fa3004bd7b35e2e8fc61c0c03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:31.096992 12716 tablet_bootstrap.cc:492] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d: Bootstrap starting.
I20260924 12:48:31.098824 12433 heartbeater.cc:503] Master 127.5.77.254:33639 was elected leader, sending a full tablet report...
I20260924 12:48:31.100979 12716 tablet_bootstrap.cc:654] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:31.105350 12715 tablet_bootstrap.cc:492] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 8c6f70afcab64a2a946d2aad7ceda99b: Bootstrap starting.
I20260924 12:48:31.102371 12716 log.cc:824] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:31.109040 12715 tablet_bootstrap.cc:654] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 8c6f70afcab64a2a946d2aad7ceda99b: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:31.110497 12715 log.cc:824] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 8c6f70afcab64a2a946d2aad7ceda99b: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:31.116122 12716 tablet_bootstrap.cc:492] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d: No bootstrap required, opened a new log
I20260924 12:48:31.116504 12716 ts_tablet_manager.cc:1397] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.004s
I20260924 12:48:31.124565 12715 tablet_bootstrap.cc:492] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 8c6f70afcab64a2a946d2aad7ceda99b: No bootstrap required, opened a new log
I20260924 12:48:31.124878 12715 ts_tablet_manager.cc:1397] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 8c6f70afcab64a2a946d2aad7ceda99b: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.010s
I20260924 12:48:31.128476 12716 raft_consensus.cc:348] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } } peers { permanent_uuid: "8c6f70afcab64a2a946d2aad7ceda99b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 39661 } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } }
I20260924 12:48:31.129138 12716 raft_consensus.cc:729] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a09a2719ee244818f474ff51937f75d, State: Initialized, Role: FOLLOWER
I20260924 12:48:31.129657 12716 consensus_queue.cc:260] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } } peers { permanent_uuid: "8c6f70afcab64a2a946d2aad7ceda99b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 39661 } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } }
I20260924 12:48:31.135731 12715 raft_consensus.cc:348] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 8c6f70afcab64a2a946d2aad7ceda99b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } } peers { permanent_uuid: "8c6f70afcab64a2a946d2aad7ceda99b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 39661 } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } }
I20260924 12:48:31.136298 12715 raft_consensus.cc:729] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 8c6f70afcab64a2a946d2aad7ceda99b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c6f70afcab64a2a946d2aad7ceda99b, State: Initialized, Role: FOLLOWER
I20260924 12:48:31.136374 12717 tablet_bootstrap.cc:492] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4b7423e914ac4794ab465c2671f93bff: Bootstrap starting.
I20260924 12:48:31.136875 12715 consensus_queue.cc:260] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 8c6f70afcab64a2a946d2aad7ceda99b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } } peers { permanent_uuid: "8c6f70afcab64a2a946d2aad7ceda99b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 39661 } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } }
I20260924 12:48:31.138783 12716 ts_tablet_manager.cc:1428] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d: Time spent starting tablet: real 0.022s user 0.014s sys 0.007s
I20260924 12:48:31.139895 12722 tablet_bootstrap.cc:492] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 0cff88768c07440e8aa9fe05f868c706: Bootstrap starting.
I20260924 12:48:31.140573 12717 tablet_bootstrap.cc:654] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4b7423e914ac4794ab465c2671f93bff: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:31.141675 12721 tablet_bootstrap.cc:492] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P a72ae1f099324d0eb8b0104438256f28: Bootstrap starting.
I20260924 12:48:31.143195 12563 heartbeater.cc:503] Master 127.5.77.254:33639 was elected leader, sending a full tablet report...
I20260924 12:48:31.145171 12715 ts_tablet_manager.cc:1428] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 8c6f70afcab64a2a946d2aad7ceda99b: Time spent starting tablet: real 0.020s user 0.014s sys 0.005s
I20260924 12:48:31.145464 12721 tablet_bootstrap.cc:654] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P a72ae1f099324d0eb8b0104438256f28: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:31.149681 12717 log.cc:824] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4b7423e914ac4794ab465c2671f93bff: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:31.150146 12722 tablet_bootstrap.cc:654] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 0cff88768c07440e8aa9fe05f868c706: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:31.151490 12722 log.cc:824] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 0cff88768c07440e8aa9fe05f868c706: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:31.153074 12721 log.cc:824] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P a72ae1f099324d0eb8b0104438256f28: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:31.153262 12717 tablet_bootstrap.cc:492] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4b7423e914ac4794ab465c2671f93bff: No bootstrap required, opened a new log
I20260924 12:48:31.153574 12717 ts_tablet_manager.cc:1397] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4b7423e914ac4794ab465c2671f93bff: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.003s
I20260924 12:48:31.153990 5431 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20260924 12:48:31.156106 12722 tablet_bootstrap.cc:492] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 0cff88768c07440e8aa9fe05f868c706: No bootstrap required, opened a new log
I20260924 12:48:31.156467 5431 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0cff88768c07440e8aa9fe05f868c706 to finish bootstrapping
I20260924 12:48:31.157135 12722 ts_tablet_manager.cc:1397] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 0cff88768c07440e8aa9fe05f868c706: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.008s
I20260924 12:48:31.158809 12721 tablet_bootstrap.cc:492] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P a72ae1f099324d0eb8b0104438256f28: No bootstrap required, opened a new log
I20260924 12:48:31.159171 12721 ts_tablet_manager.cc:1397] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P a72ae1f099324d0eb8b0104438256f28: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.000s
I20260924 12:48:31.164839 12717 raft_consensus.cc:348] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4b7423e914ac4794ab465c2671f93bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } } peers { permanent_uuid: "8c6f70afcab64a2a946d2aad7ceda99b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 39661 } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } }
I20260924 12:48:31.165447 12717 raft_consensus.cc:729] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4b7423e914ac4794ab465c2671f93bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7423e914ac4794ab465c2671f93bff, State: Initialized, Role: FOLLOWER
I20260924 12:48:31.166059 12717 consensus_queue.cc:260] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4b7423e914ac4794ab465c2671f93bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } } peers { permanent_uuid: "8c6f70afcab64a2a946d2aad7ceda99b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 39661 } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } }
I20260924 12:48:31.167906 12722 raft_consensus.cc:348] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 0cff88768c07440e8aa9fe05f868c706 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } } peers { permanent_uuid: "8c6f70afcab64a2a946d2aad7ceda99b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 39661 } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } }
I20260924 12:48:31.168438 12722 raft_consensus.cc:729] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 0cff88768c07440e8aa9fe05f868c706 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cff88768c07440e8aa9fe05f868c706, State: Initialized, Role: FOLLOWER
I20260924 12:48:31.168938 12722 consensus_queue.cc:260] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 0cff88768c07440e8aa9fe05f868c706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } } peers { permanent_uuid: "8c6f70afcab64a2a946d2aad7ceda99b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 39661 } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } }
I20260924 12:48:31.170199 12721 raft_consensus.cc:348] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P a72ae1f099324d0eb8b0104438256f28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } } peers { permanent_uuid: "8c6f70afcab64a2a946d2aad7ceda99b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 39661 } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } }
I20260924 12:48:31.170879 12721 raft_consensus.cc:729] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P a72ae1f099324d0eb8b0104438256f28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72ae1f099324d0eb8b0104438256f28, State: Initialized, Role: FOLLOWER
I20260924 12:48:31.171452 12721 consensus_queue.cc:260] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P a72ae1f099324d0eb8b0104438256f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } } peers { permanent_uuid: "8c6f70afcab64a2a946d2aad7ceda99b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 39661 } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } }
I20260924 12:48:31.177968 12722 ts_tablet_manager.cc:1428] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 0cff88768c07440e8aa9fe05f868c706: Time spent starting tablet: real 0.021s user 0.013s sys 0.006s
I20260924 12:48:31.178118 12717 ts_tablet_manager.cc:1428] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4b7423e914ac4794ab465c2671f93bff: Time spent starting tablet: real 0.024s user 0.018s sys 0.007s
I20260924 12:48:31.178750 12693 heartbeater.cc:503] Master 127.5.77.254:33639 was elected leader, sending a full tablet report...
I20260924 12:48:31.182214 12721 ts_tablet_manager.cc:1428] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P a72ae1f099324d0eb8b0104438256f28: Time spent starting tablet: real 0.023s user 0.017s sys 0.004s
I20260924 12:48:31.183247 5431 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4a09a2719ee244818f474ff51937f75d to finish bootstrapping
W20260924 12:48:31.186973 12173 tablet.cc:2369] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 0cff88768c07440e8aa9fe05f868c706: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:48:31.193715 5431 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a72ae1f099324d0eb8b0104438256f28 to finish bootstrapping
I20260924 12:48:31.202349 5431 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 8c6f70afcab64a2a946d2aad7ceda99b to finish bootstrapping
I20260924 12:48:31.210513 5431 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4b7423e914ac4794ab465c2671f93bff to finish bootstrapping
W20260924 12:48:31.219802 12694 tablet.cc:2369] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4b7423e914ac4794ab465c2671f93bff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:48:31.225019 12258 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fa53dd1fa3004bd7b35e2e8fc61c0c03"
dest_uuid: "4a09a2719ee244818f474ff51937f75d"
from {username='slave'} at 127.0.0.1:56134
I20260924 12:48:31.225360 12258 raft_consensus.cc:482] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 12:48:31.225564 12258 raft_consensus.cc:3037] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:31.227665 12258 raft_consensus.cc:504] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } } peers { permanent_uuid: "8c6f70afcab64a2a946d2aad7ceda99b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 39661 } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } }
I20260924 12:48:31.229452 12258 leader_election.cc:288] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d [CANDIDATE]: Term 1 election: Requested vote from peers a72ae1f099324d0eb8b0104438256f28 (127.5.77.195:37669), 8c6f70afcab64a2a946d2aad7ceda99b (127.5.77.196:39661), 4b7423e914ac4794ab465c2671f93bff (127.5.77.197:33815), 0cff88768c07440e8aa9fe05f868c706 (127.5.77.193:40467)
I20260924 12:48:31.241012 12518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa53dd1fa3004bd7b35e2e8fc61c0c03" candidate_uuid: "4a09a2719ee244818f474ff51937f75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8c6f70afcab64a2a946d2aad7ceda99b"
I20260924 12:48:31.241503 12518 raft_consensus.cc:3037] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 8c6f70afcab64a2a946d2aad7ceda99b [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:31.242901 12388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa53dd1fa3004bd7b35e2e8fc61c0c03" candidate_uuid: "4a09a2719ee244818f474ff51937f75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a72ae1f099324d0eb8b0104438256f28"
I20260924 12:48:31.243335 12388 raft_consensus.cc:3037] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P a72ae1f099324d0eb8b0104438256f28 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:31.244769 12518 raft_consensus.cc:2454] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 8c6f70afcab64a2a946d2aad7ceda99b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a09a2719ee244818f474ff51937f75d in term 1.
I20260924 12:48:31.245529 12648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa53dd1fa3004bd7b35e2e8fc61c0c03" candidate_uuid: "4a09a2719ee244818f474ff51937f75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4b7423e914ac4794ab465c2671f93bff"
I20260924 12:48:31.245942 12648 raft_consensus.cc:3037] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4b7423e914ac4794ab465c2671f93bff [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:31.246676 12388 raft_consensus.cc:2454] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P a72ae1f099324d0eb8b0104438256f28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a09a2719ee244818f474ff51937f75d in term 1.
I20260924 12:48:31.247345 12194 leader_election.cc:302] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 4a09a2719ee244818f474ff51937f75d, 8c6f70afcab64a2a946d2aad7ceda99b, a72ae1f099324d0eb8b0104438256f28; no voters:
I20260924 12:48:31.247838 12720 raft_consensus.cc:2781] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:31.248447 12720 raft_consensus.cc:686] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d [term 1 LEADER]: Becoming Leader. State: Replica: 4a09a2719ee244818f474ff51937f75d, State: Running, Role: LEADER
I20260924 12:48:31.249205 12648 raft_consensus.cc:2454] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4b7423e914ac4794ab465c2671f93bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a09a2719ee244818f474ff51937f75d in term 1.
I20260924 12:48:31.249086 12720 consensus_queue.cc:237] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } } peers { permanent_uuid: "8c6f70afcab64a2a946d2aad7ceda99b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 39661 } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } }
I20260924 12:48:31.249753 12127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa53dd1fa3004bd7b35e2e8fc61c0c03" candidate_uuid: "4a09a2719ee244818f474ff51937f75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0cff88768c07440e8aa9fe05f868c706"
I20260924 12:48:31.250120 12127 raft_consensus.cc:3037] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 0cff88768c07440e8aa9fe05f868c706 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:31.252681 12127 raft_consensus.cc:2454] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 0cff88768c07440e8aa9fe05f868c706 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a09a2719ee244818f474ff51937f75d in term 1.
I20260924 12:48:31.254158 11985 catalog_manager.cc:5495] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a09a2719ee244818f474ff51937f75d (127.5.77.194). New cstate: current_term: 1 leader_uuid: "4a09a2719ee244818f474ff51937f75d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c6f70afcab64a2a946d2aad7ceda99b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 39661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } health_report { overall_health: UNKNOWN } } }
W20260924 12:48:31.280212 12564 tablet.cc:2369] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 8c6f70afcab64a2a946d2aad7ceda99b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 12:48:31.328392 12434 tablet.cc:2369] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P a72ae1f099324d0eb8b0104438256f28: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 12:48:31.379133 12304 tablet.cc:2369] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:48:31.654717 12720 consensus_queue.cc:1035] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:48:31.712945 12720 consensus_queue.cc:1035] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:48:31.720927 12720 consensus_queue.cc:1035] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:48:31.721834 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 11955
I20260924 12:48:31.722114 12720 consensus_queue.cc:1035] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c6f70afcab64a2a946d2aad7ceda99b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 39661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:48:31.723320 12016 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:31.838359 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 11955
W20260924 12:48:31.855511 12693 heartbeater.cc:644] Failed to heartbeat to 127.5.77.254:33639 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.5.77.254:33639 (error 108)
W20260924 12:48:31.855779 12433 heartbeater.cc:644] Failed to heartbeat to 127.5.77.254:33639 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.5.77.254:33639 (error 108)
I20260924 12:48:31.856061 5431 raft_consensus_election-itest.cc:666] Removing tserver with uuid 8c6f70afcab64a2a946d2aad7ceda99b
W20260924 12:48:31.856125 12563 heartbeater.cc:644] Failed to heartbeat to 127.5.77.254:33639 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.5.77.254:33639 (error 108)
I20260924 12:48:31.858328 12258 consensus_queue.cc:237] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } }
I20260924 12:48:31.860348 12127 raft_consensus.cc:1261] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 0cff88768c07440e8aa9fe05f868c706 [term 1 FOLLOWER]: Refusing update from remote peer 4a09a2719ee244818f474ff51937f75d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:48:31.860361 12648 raft_consensus.cc:1261] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4b7423e914ac4794ab465c2671f93bff [term 1 FOLLOWER]: Refusing update from remote peer 4a09a2719ee244818f474ff51937f75d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:48:31.860402 12388 raft_consensus.cc:1261] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P a72ae1f099324d0eb8b0104438256f28 [term 1 FOLLOWER]: Refusing update from remote peer 4a09a2719ee244818f474ff51937f75d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:48:31.860893 12736 consensus_queue.cc:1035] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 12:48:31.861148 12742 consensus_queue.cc:1035] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 12:48:31.861384 12720 consensus_queue.cc:1035] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 12:48:31.863848 12742 raft_consensus.cc:2932] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8c6f70afcab64a2a946d2aad7ceda99b (127.5.77.196) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } } }
I20260924 12:48:31.864472 12388 raft_consensus.cc:2932] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P a72ae1f099324d0eb8b0104438256f28 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8c6f70afcab64a2a946d2aad7ceda99b (127.5.77.196) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } } }
I20260924 12:48:31.864905 12127 raft_consensus.cc:2932] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 0cff88768c07440e8aa9fe05f868c706 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8c6f70afcab64a2a946d2aad7ceda99b (127.5.77.196) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } } }
I20260924 12:48:31.875895 12648 raft_consensus.cc:2932] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4b7423e914ac4794ab465c2671f93bff [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8c6f70afcab64a2a946d2aad7ceda99b (127.5.77.196) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } } }
W20260924 12:48:31.879603 12303 heartbeater.cc:644] Failed to heartbeat to 127.5.77.254:33639 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.77.254:33639: connect: Connection refused (error 111)
W20260924 12:48:31.883524 12172 heartbeater.cc:644] Failed to heartbeat to 127.5.77.254:33639 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.77.254:33639: connect: Connection refused (error 111)
I20260924 12:48:31.901710 5431 raft_consensus_election-itest.cc:676] Pausing 3 replicas...
I20260924 12:48:31.934428 5431 raft_consensus_election-itest.cc:684] Adding back Peer 8c6f70afcab64a2a946d2aad7ceda99b and expecting timeout...
I20260924 12:48:31.936265 12258 consensus_queue.cc:237] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } } peers { permanent_uuid: "8c6f70afcab64a2a946d2aad7ceda99b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 39661 } attrs { } }
I20260924 12:48:31.938499 12518 raft_consensus.cc:1261] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 8c6f70afcab64a2a946d2aad7ceda99b [term 1 FOLLOWER]: Refusing update from remote peer 4a09a2719ee244818f474ff51937f75d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 12:48:31.939028 12736 consensus_queue.cc:1035] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c6f70afcab64a2a946d2aad7ceda99b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 39661 } attrs { }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 12:48:31.941395 12518 raft_consensus.cc:858] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 8c6f70afcab64a2a946d2aad7ceda99b [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } } peers { permanent_uuid: "8c6f70afcab64a2a946d2aad7ceda99b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 39661 } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } }.
Pending config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } }:
New config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } } peers { permanent_uuid: "8c6f70afcab64a2a946d2aad7ceda99b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 39661 } attrs { } }
W20260924 12:48:31.941517 12518 raft_consensus.cc:1565] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 8c6f70afcab64a2a946d2aad7ceda99b [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 3' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } } peers { permanent_uuid: "8c6f70afcab64a2a946d2aad7ceda99b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 39661 } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } }.
Pending config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } }:
New config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } } peers { permanent_uuid: "8c6f70afcab64a2a946d2aad7ceda99b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 39661 } attrs { } }
I20260924 12:48:31.942014 12518 raft_consensus.cc:2932] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 8c6f70afcab64a2a946d2aad7ceda99b [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8c6f70afcab64a2a946d2aad7ceda99b (127.5.77.196) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } } }
I20260924 12:48:31.946148 12518 raft_consensus.cc:1467] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 8c6f70afcab64a2a946d2aad7ceda99b [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20260924 12:48:32.036074 5431 raft_consensus_election-itest.cc:688] Timeout achieved.
I20260924 12:48:32.044548 5431 raft_consensus_election-itest.cc:699] Forcing Peer 4a09a2719ee244818f474ff51937f75d to step down...
I20260924 12:48:32.045334 12258 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fa53dd1fa3004bd7b35e2e8fc61c0c03"
dest_uuid: "4a09a2719ee244818f474ff51937f75d"
from {username='slave'} at 127.0.0.1:56134
I20260924 12:48:32.045599 12258 raft_consensus.cc:580] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d [term 1 LEADER]: Received request to step down
I20260924 12:48:32.045691 12258 raft_consensus.cc:3032] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d [term 1 LEADER]: Stepping down as leader of term 1
I20260924 12:48:32.045778 12258 raft_consensus.cc:729] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a09a2719ee244818f474ff51937f75d, State: Running, Role: LEADER
I20260924 12:48:32.046015 12258 consensus_queue.cc:260] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } } peers { permanent_uuid: "8c6f70afcab64a2a946d2aad7ceda99b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 39661 } attrs { } }
I20260924 12:48:32.046402 12258 raft_consensus.cc:3037] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:48:32.047040 5431 raft_consensus_election-itest.cc:705] Resuming Peer 0cff88768c07440e8aa9fe05f868c706 ...
I20260924 12:48:32.048156 12127 raft_consensus.cc:1261] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 0cff88768c07440e8aa9fe05f868c706 [term 1 FOLLOWER]: Refusing update from remote peer 4a09a2719ee244818f474ff51937f75d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 12:48:32.048815 5431 raft_consensus_election-itest.cc:710] Trying to elect Peer 8c6f70afcab64a2a946d2aad7ceda99b ...
I20260924 12:48:32.049463 12518 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fa53dd1fa3004bd7b35e2e8fc61c0c03"
dest_uuid: "8c6f70afcab64a2a946d2aad7ceda99b"
from {username='slave'} at 127.0.0.1:41182
I20260924 12:48:32.049722 12518 raft_consensus.cc:482] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 8c6f70afcab64a2a946d2aad7ceda99b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 12:48:32.049853 12518 raft_consensus.cc:3037] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 8c6f70afcab64a2a946d2aad7ceda99b [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:48:32.051844 12518 raft_consensus.cc:504] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 8c6f70afcab64a2a946d2aad7ceda99b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } } peers { permanent_uuid: "8c6f70afcab64a2a946d2aad7ceda99b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 39661 } attrs { } }
I20260924 12:48:32.053442 12518 leader_election.cc:288] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 8c6f70afcab64a2a946d2aad7ceda99b [CANDIDATE]: Term 2 election: Requested vote from peers 4a09a2719ee244818f474ff51937f75d (127.5.77.194:46345), a72ae1f099324d0eb8b0104438256f28 (127.5.77.195:37669), 4b7423e914ac4794ab465c2671f93bff (127.5.77.197:33815), 0cff88768c07440e8aa9fe05f868c706 (127.5.77.193:40467)
I20260924 12:48:32.057528 5431 cluster_itest_util.cc:256] Not converged past 4 yet: 1.2 1.3 1.3
I20260924 12:48:32.061600 12258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa53dd1fa3004bd7b35e2e8fc61c0c03" candidate_uuid: "8c6f70afcab64a2a946d2aad7ceda99b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4a09a2719ee244818f474ff51937f75d"
I20260924 12:48:32.062414 12127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa53dd1fa3004bd7b35e2e8fc61c0c03" candidate_uuid: "8c6f70afcab64a2a946d2aad7ceda99b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0cff88768c07440e8aa9fe05f868c706"
I20260924 12:48:32.062647 12127 raft_consensus.cc:1799] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 0cff88768c07440e8aa9fe05f868c706 [term 1 FOLLOWER]: Handling vote request from an unknown peer 8c6f70afcab64a2a946d2aad7ceda99b
I20260924 12:48:32.062767 12127 raft_consensus.cc:3037] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 0cff88768c07440e8aa9fe05f868c706 [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:48:32.064648 12258 raft_consensus.cc:2454] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c6f70afcab64a2a946d2aad7ceda99b in term 2.
I20260924 12:48:32.065362 12127 raft_consensus.cc:2454] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 0cff88768c07440e8aa9fe05f868c706 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c6f70afcab64a2a946d2aad7ceda99b in term 2.
I20260924 12:48:32.065713 12451 leader_election.cc:302] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 8c6f70afcab64a2a946d2aad7ceda99b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 0cff88768c07440e8aa9fe05f868c706, 4a09a2719ee244818f474ff51937f75d, 8c6f70afcab64a2a946d2aad7ceda99b; no voters:
I20260924 12:48:32.066092 12746 raft_consensus.cc:2781] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 8c6f70afcab64a2a946d2aad7ceda99b [term 2 FOLLOWER]: Leader election won for term 2
I20260924 12:48:32.066632 12746 raft_consensus.cc:686] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 8c6f70afcab64a2a946d2aad7ceda99b [term 2 LEADER]: Becoming Leader. State: Replica: 8c6f70afcab64a2a946d2aad7ceda99b, State: Running, Role: LEADER
I20260924 12:48:32.067206 12746 consensus_queue.cc:237] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 8c6f70afcab64a2a946d2aad7ceda99b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } } peers { permanent_uuid: "8c6f70afcab64a2a946d2aad7ceda99b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 39661 } attrs { } }
I20260924 12:48:32.161165 5431 cluster_itest_util.cc:256] Not converged past 4 yet: 1.2 1.3 2.4
I20260924 12:48:32.364652 5431 cluster_itest_util.cc:256] Not converged past 4 yet: 1.2 1.3 2.4
I20260924 12:48:32.592758 12127 raft_consensus.cc:1261] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 0cff88768c07440e8aa9fe05f868c706 [term 2 FOLLOWER]: Refusing update from remote peer 8c6f70afcab64a2a946d2aad7ceda99b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260924 12:48:32.593439 12746 consensus_queue.cc:1035] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 8c6f70afcab64a2a946d2aad7ceda99b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 12:48:32.657344 12258 raft_consensus.cc:1261] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d [term 2 FOLLOWER]: Refusing update from remote peer 8c6f70afcab64a2a946d2aad7ceda99b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260924 12:48:32.658119 12746 consensus_queue.cc:1035] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 8c6f70afcab64a2a946d2aad7ceda99b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 12:48:32.661134 12772 raft_consensus.cc:2932] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 8c6f70afcab64a2a946d2aad7ceda99b [term 2 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8c6f70afcab64a2a946d2aad7ceda99b (127.5.77.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } } peers { permanent_uuid: "8c6f70afcab64a2a946d2aad7ceda99b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 39661 } attrs { } } }
I20260924 12:48:32.661703 12258 raft_consensus.cc:2932] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4a09a2719ee244818f474ff51937f75d [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8c6f70afcab64a2a946d2aad7ceda99b (127.5.77.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } } peers { permanent_uuid: "8c6f70afcab64a2a946d2aad7ceda99b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 39661 } attrs { } } }
W20260924 12:48:32.664332 12258 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:56134 (request call id 47) took 728 ms (client timeout 100 ms). Trace:
I20260924 12:48:32.665285 12127 raft_consensus.cc:2932] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 0cff88768c07440e8aa9fe05f868c706 [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8c6f70afcab64a2a946d2aad7ceda99b (127.5.77.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } } peers { permanent_uuid: "8c6f70afcab64a2a946d2aad7ceda99b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 39661 } attrs { } } }
W20260924 12:48:32.664489 12258 rpcz_store.cc:269] 0924 12:48:31.935380 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:48:31.935471 (+ 91us) service_pool.cc:227] Handling call
0924 12:48:31.937287 (+ 1816us) log.cc:842] Serialized 600 byte log entry
0924 12:48:32.664260 (+726973us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":487,"raft.queue_time_us":872,"raft.run_cpu_time_us":639,"raft.run_wall_time_us":642,"spinlock_wait_cycles":41856,"thread_start_us":336,"threads_started":1}
I20260924 12:48:32.671159 5431 raft_consensus_election-itest.cc:716] Resuming remaining nodes...
I20260924 12:48:32.673130 12388 raft_consensus.cc:1261] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P a72ae1f099324d0eb8b0104438256f28 [term 1 FOLLOWER]: Refusing update from remote peer 4a09a2719ee244818f474ff51937f75d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 12:48:32.675537 12648 raft_consensus.cc:1261] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4b7423e914ac4794ab465c2671f93bff [term 1 FOLLOWER]: Refusing update from remote peer 4a09a2719ee244818f474ff51937f75d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 12:48:32.682086 12388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa53dd1fa3004bd7b35e2e8fc61c0c03" candidate_uuid: "8c6f70afcab64a2a946d2aad7ceda99b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a72ae1f099324d0eb8b0104438256f28"
I20260924 12:48:32.682312 12388 raft_consensus.cc:1799] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P a72ae1f099324d0eb8b0104438256f28 [term 1 FOLLOWER]: Handling vote request from an unknown peer 8c6f70afcab64a2a946d2aad7ceda99b
I20260924 12:48:32.682418 12388 raft_consensus.cc:3037] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P a72ae1f099324d0eb8b0104438256f28 [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:48:32.685323 12388 raft_consensus.cc:2454] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P a72ae1f099324d0eb8b0104438256f28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c6f70afcab64a2a946d2aad7ceda99b in term 2.
I20260924 12:48:32.685746 12647 raft_consensus.cc:3037] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4b7423e914ac4794ab465c2671f93bff [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:48:32.685997 12387 raft_consensus.cc:1261] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P a72ae1f099324d0eb8b0104438256f28 [term 2 FOLLOWER]: Refusing update from remote peer 8c6f70afcab64a2a946d2aad7ceda99b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260924 12:48:32.686661 12648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa53dd1fa3004bd7b35e2e8fc61c0c03" candidate_uuid: "8c6f70afcab64a2a946d2aad7ceda99b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4b7423e914ac4794ab465c2671f93bff"
I20260924 12:48:32.686828 12772 consensus_queue.cc:1035] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 8c6f70afcab64a2a946d2aad7ceda99b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 12:48:32.689047 12647 raft_consensus.cc:1261] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4b7423e914ac4794ab465c2671f93bff [term 2 FOLLOWER]: Refusing update from remote peer 8c6f70afcab64a2a946d2aad7ceda99b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260924 12:48:32.690852 12746 consensus_queue.cc:1035] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 8c6f70afcab64a2a946d2aad7ceda99b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 12:48:32.691468 12648 raft_consensus.cc:1799] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4b7423e914ac4794ab465c2671f93bff [term 2 FOLLOWER]: Handling vote request from an unknown peer 8c6f70afcab64a2a946d2aad7ceda99b
I20260924 12:48:32.693776 12648 raft_consensus.cc:2454] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4b7423e914ac4794ab465c2671f93bff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c6f70afcab64a2a946d2aad7ceda99b in term 2.
I20260924 12:48:32.695019 12387 raft_consensus.cc:2932] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P a72ae1f099324d0eb8b0104438256f28 [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8c6f70afcab64a2a946d2aad7ceda99b (127.5.77.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } } peers { permanent_uuid: "8c6f70afcab64a2a946d2aad7ceda99b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 39661 } attrs { } } }
I20260924 12:48:32.712489 12647 raft_consensus.cc:2932] T fa53dd1fa3004bd7b35e2e8fc61c0c03 P 4b7423e914ac4794ab465c2671f93bff [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8c6f70afcab64a2a946d2aad7ceda99b (127.5.77.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a09a2719ee244818f474ff51937f75d" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 46345 } } peers { permanent_uuid: "a72ae1f099324d0eb8b0104438256f28" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 37669 } } peers { permanent_uuid: "4b7423e914ac4794ab465c2671f93bff" member_type: VOTER last_known_addr { host: "127.5.77.197" port: 33815 } } peers { permanent_uuid: "0cff88768c07440e8aa9fe05f868c706" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 40467 } } peers { permanent_uuid: "8c6f70afcab64a2a946d2aad7ceda99b" member_type: VOTER last_known_addr { host: "127.5.77.196" port: 39661 } attrs { } } }
I20260924 12:48:32.719568 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 12045
I20260924 12:48:32.720610 12167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:32.806860 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 12045
I20260924 12:48:32.820098 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 12176
I20260924 12:48:32.821148 12298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:32.913925 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 12176
I20260924 12:48:32.928013 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 12306
I20260924 12:48:32.929016 12428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:33.015913 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 12306
I20260924 12:48:33.028417 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 12436
I20260924 12:48:33.029506 12558 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:33.118897 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 12436
I20260924 12:48:33.131800 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 12566
I20260924 12:48:33.132898 12688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:33.220865 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 12566
2026-09-24T12:48:33Z chronyd exiting
[ OK ] RaftConsensusElectionITest.ElectPendingVoter (5069 ms)
[ RUN ] RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy
I20260924 12:48:33.254969 5431 test_util.cc:268] Using random seed: 75132434
I20260924 12:48:33.256531 5431 ts_itest-base.cc:112] Starting cluster with:
I20260924 12:48:33.256604 5431 ts_itest-base.cc:113] --------------
I20260924 12:48:33.256639 5431 ts_itest-base.cc:114] 3 tablet servers
I20260924 12:48:33.256688 5431 ts_itest-base.cc:115] 3 replicas per TS
I20260924 12:48:33.256722 5431 ts_itest-base.cc:116] --------------
2026-09-24T12:48:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:48:33Z Disabled control of system clock
I20260924 12:48:33.285744 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.77.254:38013
--webserver_interface=127.5.77.254
--webserver_port=0
--builtin_ntp_servers=127.5.77.212:35701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.77.254:38013
--master_add_server_when_underreplicated=false
--raft_prepare_replacement_before_eviction=false with env {}
W20260924 12:48:33.600255 12802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:33.600541 12802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:33.600652 12802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:33.607919 12802 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260924 12:48:33.608043 12802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:48:33.608108 12802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:33.608186 12802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:48:33.608242 12802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:48:33.617471 12802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:35701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
--master_add_server_when_underreplicated=false
--ipki_ca_key_size=768
--master_addresses=127.5.77.254:38013
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.77.254:38013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.77.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:33.618808 12802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:33.620592 12802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:33.631636 12807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:33.631592 12810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:33.631636 12808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:33.632359 12802 server_base.cc:1034] running on GCE node
I20260924 12:48:33.633155 12802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:33.634550 12802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:33.635769 12802 hybrid_clock.cc:648] HybridClock initialized: now 1790254113635711 us; error 51 us; skew 500 ppm
I20260924 12:48:33.636070 12802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:33.638192 12802 webserver.cc:460] Webserver started at http://127.5.77.254:43969/ using document root <none> and password file <none>
I20260924 12:48:33.638883 12802 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:33.638967 12802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:33.639210 12802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:33.641928 12802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "eb6156389fe548ceb292b29fa38d42f2"
format_stamp: "Formatted at 2026-09-24 12:48:33 on dist-test-slave-b8q3"
I20260924 12:48:33.642803 12802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "eb6156389fe548ceb292b29fa38d42f2"
format_stamp: "Formatted at 2026-09-24 12:48:33 on dist-test-slave-b8q3"
I20260924 12:48:33.647928 12802 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 12:48:33.651460 12816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:33.653081 12802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 12:48:33.653259 12802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
uuid: "eb6156389fe548ceb292b29fa38d42f2"
format_stamp: "Formatted at 2026-09-24 12:48:33 on dist-test-slave-b8q3"
I20260924 12:48:33.653470 12802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:33.682920 12802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:33.683811 12802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:33.684068 12802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:33.700542 12802 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.254:38013
I20260924 12:48:33.700567 12867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.254:38013 every 8 connection(s)
I20260924 12:48:33.701830 12802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260924 12:48:33.704124 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 12802
I20260924 12:48:33.704366 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260924 12:48:33.706488 12868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:33.718448 12868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2: Bootstrap starting.
I20260924 12:48:33.721714 12868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:33.722721 12868 log.cc:824] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:33.725366 12868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2: No bootstrap required, opened a new log
I20260924 12:48:33.732661 12868 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6156389fe548ceb292b29fa38d42f2" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 38013 } }
I20260924 12:48:33.733027 12868 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:48:33.733103 12868 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6156389fe548ceb292b29fa38d42f2, State: Initialized, Role: FOLLOWER
I20260924 12:48:33.733633 12868 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6156389fe548ceb292b29fa38d42f2" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 38013 } }
I20260924 12:48:33.733829 12868 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:48:33.733927 12868 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:48:33.734046 12868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:33.736431 12868 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6156389fe548ceb292b29fa38d42f2" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 38013 } }
I20260924 12:48:33.736877 12868 leader_election.cc:302] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb6156389fe548ceb292b29fa38d42f2; no voters:
I20260924 12:48:33.737381 12868 leader_election.cc:288] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:48:33.737500 12873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:33.738157 12873 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [term 1 LEADER]: Becoming Leader. State: Replica: eb6156389fe548ceb292b29fa38d42f2, State: Running, Role: LEADER
I20260924 12:48:33.738750 12873 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6156389fe548ceb292b29fa38d42f2" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 38013 } }
I20260924 12:48:33.739289 12868 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:48:33.741588 12874 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb6156389fe548ceb292b29fa38d42f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6156389fe548ceb292b29fa38d42f2" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 38013 } } }
I20260924 12:48:33.741652 12875 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb6156389fe548ceb292b29fa38d42f2. Latest consensus state: current_term: 1 leader_uuid: "eb6156389fe548ceb292b29fa38d42f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6156389fe548ceb292b29fa38d42f2" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 38013 } } }
I20260924 12:48:33.741966 12874 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [sys.catalog]: This master's current role is: LEADER
I20260924 12:48:33.742017 12875 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [sys.catalog]: This master's current role is: LEADER
I20260924 12:48:33.743830 12882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:48:33.747970 12882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:48:33.756767 12882 catalog_manager.cc:1347] Generated new cluster ID: 01d317af27644887acb13ae3348c26c6
I20260924 12:48:33.756883 12882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:48:33.787611 12882 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:48:33.788901 12882 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:48:33.795621 12882 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2: Generated new TSK 0
I20260924 12:48:33.796445 12882 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:48:33.812484 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.193:0
--local_ip_for_outbound_sockets=127.5.77.193
--webserver_interface=127.5.77.193
--webserver_port=0
--tserver_master_addrs=127.5.77.254:38013
--builtin_ntp_servers=127.5.77.212:35701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
W20260924 12:48:34.136188 12892 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20260924 12:48:34.136533 12892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:34.136619 12892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:34.136741 12892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:34.144291 12892 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20260924 12:48:34.144402 12892 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20260924 12:48:34.144472 12892 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260924 12:48:34.144550 12892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:34.144711 12892 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20260924 12:48:34.144775 12892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.193
I20260924 12:48:34.154102 12892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:35701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.77.193
--webserver_port=0
--tserver_master_addrs=127.5.77.254:38013
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.193
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:34.155467 12892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:34.157241 12892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:34.170080 12897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:34.170054 12898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:34.170111 12900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:34.171947 12892 server_base.cc:1034] running on GCE node
I20260924 12:48:34.172544 12892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:34.173461 12892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:34.174664 12892 hybrid_clock.cc:648] HybridClock initialized: now 1790254114174614 us; error 45 us; skew 500 ppm
I20260924 12:48:34.174966 12892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:34.182767 12892 webserver.cc:460] Webserver started at http://127.5.77.193:32777/ using document root <none> and password file <none>
I20260924 12:48:34.183495 12892 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:34.183585 12892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:34.183905 12892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:34.186537 12892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "84b1720d1c674777bd8a4d94314a4c77"
format_stamp: "Formatted at 2026-09-24 12:48:34 on dist-test-slave-b8q3"
I20260924 12:48:34.187382 12892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "84b1720d1c674777bd8a4d94314a4c77"
format_stamp: "Formatted at 2026-09-24 12:48:34 on dist-test-slave-b8q3"
I20260924 12:48:34.192462 12892 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 12:48:34.195851 12906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:34.197281 12892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260924 12:48:34.197443 12892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "84b1720d1c674777bd8a4d94314a4c77"
format_stamp: "Formatted at 2026-09-24 12:48:34 on dist-test-slave-b8q3"
I20260924 12:48:34.197664 12892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:34.221655 12892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:34.222558 12892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:34.222828 12892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:34.223731 12892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:34.225325 12892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:48:34.225407 12892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:34.225486 12892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:48:34.225545 12892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:34.254809 12892 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.193:46525
I20260924 12:48:34.254885 13018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.193:46525 every 8 connection(s)
I20260924 12:48:34.256156 12892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260924 12:48:34.265471 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 12892
I20260924 12:48:34.265667 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260924 12:48:34.269160 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.194:0
--local_ip_for_outbound_sockets=127.5.77.194
--webserver_interface=127.5.77.194
--webserver_port=0
--tserver_master_addrs=127.5.77.254:38013
--builtin_ntp_servers=127.5.77.212:35701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20260924 12:48:34.269855 13019 heartbeater.cc:347] Connected to a master server at 127.5.77.254:38013
I20260924 12:48:34.270120 13019 heartbeater.cc:464] Registering TS with master...
I20260924 12:48:34.270784 13019 heartbeater.cc:511] Master 127.5.77.254:38013 requested a full tablet report, sending...
I20260924 12:48:34.272789 12833 ts_manager.cc:194] Registered new tserver with Master: 84b1720d1c674777bd8a4d94314a4c77 (127.5.77.193:46525)
I20260924 12:48:34.274644 12833 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.193:32985
W20260924 12:48:34.581385 13023 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20260924 12:48:34.581717 13023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:34.581779 13023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:34.581853 13023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:34.589193 13023 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20260924 12:48:34.589283 13023 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20260924 12:48:34.589346 13023 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260924 12:48:34.589398 13023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:34.589509 13023 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20260924 12:48:34.589557 13023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.194
I20260924 12:48:34.598748 13023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:35701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.77.194
--webserver_port=0
--tserver_master_addrs=127.5.77.254:38013
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.194
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:34.599936 13023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:34.601722 13023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:34.614730 13029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:34.614791 13031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:34.614758 13028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:34.615787 13023 server_base.cc:1034] running on GCE node
I20260924 12:48:34.616355 13023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:34.617261 13023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:34.618474 13023 hybrid_clock.cc:648] HybridClock initialized: now 1790254114618408 us; error 45 us; skew 500 ppm
I20260924 12:48:34.618795 13023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:34.620862 13023 webserver.cc:460] Webserver started at http://127.5.77.194:38887/ using document root <none> and password file <none>
I20260924 12:48:34.621546 13023 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:34.621675 13023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:34.621959 13023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:34.624552 13023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "229859ec43934d368d51025f812bb247"
format_stamp: "Formatted at 2026-09-24 12:48:34 on dist-test-slave-b8q3"
I20260924 12:48:34.625360 13023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "229859ec43934d368d51025f812bb247"
format_stamp: "Formatted at 2026-09-24 12:48:34 on dist-test-slave-b8q3"
I20260924 12:48:34.630371 13023 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 12:48:34.633723 13037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:34.635145 13023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:48:34.635298 13023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "229859ec43934d368d51025f812bb247"
format_stamp: "Formatted at 2026-09-24 12:48:34 on dist-test-slave-b8q3"
I20260924 12:48:34.635501 13023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:34.664517 13023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:34.665388 13023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:34.665655 13023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:34.666597 13023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:34.668025 13023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:48:34.668118 13023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:34.668226 13023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:48:34.668293 13023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:34.697312 13023 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.194:39323
I20260924 12:48:34.697366 13149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.194:39323 every 8 connection(s)
I20260924 12:48:34.698582 13023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260924 12:48:34.700399 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 13023
I20260924 12:48:34.700560 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260924 12:48:34.704027 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.195:0
--local_ip_for_outbound_sockets=127.5.77.195
--webserver_interface=127.5.77.195
--webserver_port=0
--tserver_master_addrs=127.5.77.254:38013
--builtin_ntp_servers=127.5.77.212:35701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20260924 12:48:34.714570 13150 heartbeater.cc:347] Connected to a master server at 127.5.77.254:38013
I20260924 12:48:34.714886 13150 heartbeater.cc:464] Registering TS with master...
I20260924 12:48:34.715514 13150 heartbeater.cc:511] Master 127.5.77.254:38013 requested a full tablet report, sending...
I20260924 12:48:34.716797 12833 ts_manager.cc:194] Registered new tserver with Master: 229859ec43934d368d51025f812bb247 (127.5.77.194:39323)
I20260924 12:48:34.717620 12833 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.194:39813
W20260924 12:48:35.028162 13153 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20260924 12:48:35.028501 13153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:35.028651 13153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:35.028769 13153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:35.036228 13153 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20260924 12:48:35.036326 13153 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20260924 12:48:35.036389 13153 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260924 12:48:35.036441 13153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:35.036551 13153 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20260924 12:48:35.036609 13153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.195
I20260924 12:48:35.046214 13153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:35701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.77.195
--webserver_port=0
--tserver_master_addrs=127.5.77.254:38013
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.195
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:35.047511 13153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:35.049278 13153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:35.062732 13161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:35.062986 13158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:35.062747 13159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:35.063210 13153 server_base.cc:1034] running on GCE node
I20260924 12:48:35.064028 13153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:35.064952 13153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:35.066140 13153 hybrid_clock.cc:648] HybridClock initialized: now 1790254115066077 us; error 56 us; skew 500 ppm
I20260924 12:48:35.066504 13153 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:35.068980 13153 webserver.cc:460] Webserver started at http://127.5.77.195:36971/ using document root <none> and password file <none>
I20260924 12:48:35.069729 13153 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:35.069825 13153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:35.070140 13153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:35.072817 13153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "eb5cc48c49024c098a500461c6595b05"
format_stamp: "Formatted at 2026-09-24 12:48:35 on dist-test-slave-b8q3"
I20260924 12:48:35.073567 13153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "eb5cc48c49024c098a500461c6595b05"
format_stamp: "Formatted at 2026-09-24 12:48:35 on dist-test-slave-b8q3"
I20260924 12:48:35.079001 13153 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260924 12:48:35.082653 13167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:35.084164 13153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:48:35.084363 13153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "eb5cc48c49024c098a500461c6595b05"
format_stamp: "Formatted at 2026-09-24 12:48:35 on dist-test-slave-b8q3"
I20260924 12:48:35.084575 13153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:35.099895 13153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:35.100838 13153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:35.101105 13153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:35.102092 13153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:35.103554 13153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:48:35.103649 13153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:35.103756 13153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:48:35.103822 13153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:35.134101 13153 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.195:42825
I20260924 12:48:35.134172 13279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.195:42825 every 8 connection(s)
I20260924 12:48:35.135478 13153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260924 12:48:35.138494 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 13153
I20260924 12:48:35.138666 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260924 12:48:35.149821 13280 heartbeater.cc:347] Connected to a master server at 127.5.77.254:38013
I20260924 12:48:35.150131 13280 heartbeater.cc:464] Registering TS with master...
I20260924 12:48:35.150859 13280 heartbeater.cc:511] Master 127.5.77.254:38013 requested a full tablet report, sending...
I20260924 12:48:35.152199 12833 ts_manager.cc:194] Registered new tserver with Master: eb5cc48c49024c098a500461c6595b05 (127.5.77.195:42825)
I20260924 12:48:35.153008 12833 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.195:48957
I20260924 12:48:35.154399 5431 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:48:35.169225 12833 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49948:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20260924 12:48:35.224376 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 6796266c3d914a69945997c45a5e42fb (DEFAULT_TABLE table=TestTable [id=a7ee2e5f53fc49ddb5df3764f131bcd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:48:35.224494 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 6796266c3d914a69945997c45a5e42fb (DEFAULT_TABLE table=TestTable [id=a7ee2e5f53fc49ddb5df3764f131bcd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:48:35.224735 13215 tablet_service.cc:1467] Processing CreateTablet for tablet 6796266c3d914a69945997c45a5e42fb (DEFAULT_TABLE table=TestTable [id=a7ee2e5f53fc49ddb5df3764f131bcd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:48:35.225684 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6796266c3d914a69945997c45a5e42fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:35.225836 13215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6796266c3d914a69945997c45a5e42fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:35.225836 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6796266c3d914a69945997c45a5e42fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:35.236496 13298 tablet_bootstrap.cc:492] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77: Bootstrap starting.
I20260924 12:48:35.241101 13298 tablet_bootstrap.cc:654] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:35.242197 13298 log.cc:824] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:35.244803 13300 tablet_bootstrap.cc:492] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: Bootstrap starting.
I20260924 12:48:35.245445 13298 tablet_bootstrap.cc:492] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77: No bootstrap required, opened a new log
I20260924 12:48:35.245713 13298 ts_tablet_manager.cc:1397] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 12:48:35.247646 13300 tablet_bootstrap.cc:654] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:35.248293 13299 tablet_bootstrap.cc:492] T 6796266c3d914a69945997c45a5e42fb P 229859ec43934d368d51025f812bb247: Bootstrap starting.
I20260924 12:48:35.248867 13300 log.cc:824] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:35.251658 13299 tablet_bootstrap.cc:654] T 6796266c3d914a69945997c45a5e42fb P 229859ec43934d368d51025f812bb247: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:35.251900 13300 tablet_bootstrap.cc:492] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: No bootstrap required, opened a new log
I20260924 12:48:35.252177 13300 ts_tablet_manager.cc:1397] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260924 12:48:35.252902 13299 log.cc:824] T 6796266c3d914a69945997c45a5e42fb P 229859ec43934d368d51025f812bb247: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:35.254060 13298 raft_consensus.cc:348] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229859ec43934d368d51025f812bb247" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39323 } } peers { permanent_uuid: "84b1720d1c674777bd8a4d94314a4c77" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46525 } } peers { permanent_uuid: "eb5cc48c49024c098a500461c6595b05" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 42825 } }
I20260924 12:48:35.254691 13298 raft_consensus.cc:374] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:48:35.254904 13298 raft_consensus.cc:729] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84b1720d1c674777bd8a4d94314a4c77, State: Initialized, Role: FOLLOWER
I20260924 12:48:35.255867 13299 tablet_bootstrap.cc:492] T 6796266c3d914a69945997c45a5e42fb P 229859ec43934d368d51025f812bb247: No bootstrap required, opened a new log
I20260924 12:48:35.255684 13298 consensus_queue.cc:260] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229859ec43934d368d51025f812bb247" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39323 } } peers { permanent_uuid: "84b1720d1c674777bd8a4d94314a4c77" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46525 } } peers { permanent_uuid: "eb5cc48c49024c098a500461c6595b05" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 42825 } }
I20260924 12:48:35.256274 13299 ts_tablet_manager.cc:1397] T 6796266c3d914a69945997c45a5e42fb P 229859ec43934d368d51025f812bb247: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.005s
I20260924 12:48:35.262913 13300 raft_consensus.cc:348] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229859ec43934d368d51025f812bb247" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39323 } } peers { permanent_uuid: "84b1720d1c674777bd8a4d94314a4c77" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46525 } } peers { permanent_uuid: "eb5cc48c49024c098a500461c6595b05" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 42825 } }
I20260924 12:48:35.263497 13300 raft_consensus.cc:374] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:48:35.263631 13300 raft_consensus.cc:729] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb5cc48c49024c098a500461c6595b05, State: Initialized, Role: FOLLOWER
I20260924 12:48:35.264426 13300 consensus_queue.cc:260] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229859ec43934d368d51025f812bb247" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39323 } } peers { permanent_uuid: "84b1720d1c674777bd8a4d94314a4c77" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46525 } } peers { permanent_uuid: "eb5cc48c49024c098a500461c6595b05" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 42825 } }
I20260924 12:48:35.266607 13299 raft_consensus.cc:348] T 6796266c3d914a69945997c45a5e42fb P 229859ec43934d368d51025f812bb247 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229859ec43934d368d51025f812bb247" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39323 } } peers { permanent_uuid: "84b1720d1c674777bd8a4d94314a4c77" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46525 } } peers { permanent_uuid: "eb5cc48c49024c098a500461c6595b05" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 42825 } }
I20260924 12:48:35.267786 13299 raft_consensus.cc:374] T 6796266c3d914a69945997c45a5e42fb P 229859ec43934d368d51025f812bb247 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:48:35.267869 13019 heartbeater.cc:503] Master 127.5.77.254:38013 was elected leader, sending a full tablet report...
I20260924 12:48:35.267949 13299 raft_consensus.cc:729] T 6796266c3d914a69945997c45a5e42fb P 229859ec43934d368d51025f812bb247 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 229859ec43934d368d51025f812bb247, State: Initialized, Role: FOLLOWER
I20260924 12:48:35.268719 13299 consensus_queue.cc:260] T 6796266c3d914a69945997c45a5e42fb P 229859ec43934d368d51025f812bb247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229859ec43934d368d51025f812bb247" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39323 } } peers { permanent_uuid: "84b1720d1c674777bd8a4d94314a4c77" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46525 } } peers { permanent_uuid: "eb5cc48c49024c098a500461c6595b05" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 42825 } }
I20260924 12:48:35.272686 13300 ts_tablet_manager.cc:1428] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: Time spent starting tablet: real 0.020s user 0.009s sys 0.009s
I20260924 12:48:35.273078 13280 heartbeater.cc:503] Master 127.5.77.254:38013 was elected leader, sending a full tablet report...
I20260924 12:48:35.273483 13298 ts_tablet_manager.cc:1428] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77: Time spent starting tablet: real 0.028s user 0.017s sys 0.004s
I20260924 12:48:35.274358 13150 heartbeater.cc:503] Master 127.5.77.254:38013 was elected leader, sending a full tablet report...
I20260924 12:48:35.274924 13299 ts_tablet_manager.cc:1428] T 6796266c3d914a69945997c45a5e42fb P 229859ec43934d368d51025f812bb247: Time spent starting tablet: real 0.018s user 0.014s sys 0.003s
W20260924 12:48:35.300999 13151 tablet.cc:2369] T 6796266c3d914a69945997c45a5e42fb P 229859ec43934d368d51025f812bb247: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 12:48:35.337430 13281 tablet.cc:2369] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 12:48:35.359445 13020 tablet.cc:2369] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:48:35.536345 13304 raft_consensus.cc:482] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:48:35.536613 13304 raft_consensus.cc:504] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229859ec43934d368d51025f812bb247" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39323 } } peers { permanent_uuid: "84b1720d1c674777bd8a4d94314a4c77" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46525 } } peers { permanent_uuid: "eb5cc48c49024c098a500461c6595b05" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 42825 } }
I20260924 12:48:35.538213 13304 leader_election.cc:288] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 229859ec43934d368d51025f812bb247 (127.5.77.194:39323), eb5cc48c49024c098a500461c6595b05 (127.5.77.195:42825)
I20260924 12:48:35.547631 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796266c3d914a69945997c45a5e42fb" candidate_uuid: "84b1720d1c674777bd8a4d94314a4c77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "229859ec43934d368d51025f812bb247" is_pre_election: true
I20260924 12:48:35.547830 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796266c3d914a69945997c45a5e42fb" candidate_uuid: "84b1720d1c674777bd8a4d94314a4c77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb5cc48c49024c098a500461c6595b05" is_pre_election: true
I20260924 12:48:35.548168 13105 raft_consensus.cc:2454] T 6796266c3d914a69945997c45a5e42fb P 229859ec43934d368d51025f812bb247 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84b1720d1c674777bd8a4d94314a4c77 in term 0.
I20260924 12:48:35.548285 13235 raft_consensus.cc:2454] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84b1720d1c674777bd8a4d94314a4c77 in term 0.
I20260924 12:48:35.548815 12910 leader_election.cc:302] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 229859ec43934d368d51025f812bb247, 84b1720d1c674777bd8a4d94314a4c77, eb5cc48c49024c098a500461c6595b05; no voters:
I20260924 12:48:35.549202 13304 raft_consensus.cc:2781] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:48:35.549345 13304 raft_consensus.cc:482] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:48:35.549438 13304 raft_consensus.cc:3037] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:35.551465 13304 raft_consensus.cc:504] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229859ec43934d368d51025f812bb247" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39323 } } peers { permanent_uuid: "84b1720d1c674777bd8a4d94314a4c77" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46525 } } peers { permanent_uuid: "eb5cc48c49024c098a500461c6595b05" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 42825 } }
I20260924 12:48:35.552054 13304 leader_election.cc:288] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [CANDIDATE]: Term 1 election: Requested vote from peers 229859ec43934d368d51025f812bb247 (127.5.77.194:39323), eb5cc48c49024c098a500461c6595b05 (127.5.77.195:42825)
I20260924 12:48:35.552513 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796266c3d914a69945997c45a5e42fb" candidate_uuid: "84b1720d1c674777bd8a4d94314a4c77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "229859ec43934d368d51025f812bb247"
I20260924 12:48:35.552534 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796266c3d914a69945997c45a5e42fb" candidate_uuid: "84b1720d1c674777bd8a4d94314a4c77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb5cc48c49024c098a500461c6595b05"
I20260924 12:48:35.552709 13105 raft_consensus.cc:3037] T 6796266c3d914a69945997c45a5e42fb P 229859ec43934d368d51025f812bb247 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:35.552706 13235 raft_consensus.cc:3037] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:35.554960 13235 raft_consensus.cc:2454] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84b1720d1c674777bd8a4d94314a4c77 in term 1.
I20260924 12:48:35.555009 13105 raft_consensus.cc:2454] T 6796266c3d914a69945997c45a5e42fb P 229859ec43934d368d51025f812bb247 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84b1720d1c674777bd8a4d94314a4c77 in term 1.
I20260924 12:48:35.555423 12908 leader_election.cc:302] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84b1720d1c674777bd8a4d94314a4c77, eb5cc48c49024c098a500461c6595b05; no voters:
I20260924 12:48:35.555725 13304 raft_consensus.cc:2781] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:35.556102 13304 raft_consensus.cc:686] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [term 1 LEADER]: Becoming Leader. State: Replica: 84b1720d1c674777bd8a4d94314a4c77, State: Running, Role: LEADER
I20260924 12:48:35.556567 13304 consensus_queue.cc:237] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229859ec43934d368d51025f812bb247" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39323 } } peers { permanent_uuid: "84b1720d1c674777bd8a4d94314a4c77" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46525 } } peers { permanent_uuid: "eb5cc48c49024c098a500461c6595b05" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 42825 } }
I20260924 12:48:35.561167 12833 catalog_manager.cc:5495] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84b1720d1c674777bd8a4d94314a4c77 (127.5.77.193). New cstate: current_term: 1 leader_uuid: "84b1720d1c674777bd8a4d94314a4c77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229859ec43934d368d51025f812bb247" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39323 } } peers { permanent_uuid: "84b1720d1c674777bd8a4d94314a4c77" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46525 } } peers { permanent_uuid: "eb5cc48c49024c098a500461c6595b05" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 42825 } } }
I20260924 12:48:35.648214 5431 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:48:35.650192 5431 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 84b1720d1c674777bd8a4d94314a4c77 to finish bootstrapping
I20260924 12:48:35.658747 5431 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 229859ec43934d368d51025f812bb247 to finish bootstrapping
I20260924 12:48:35.666858 5431 ts_itest-base.cc:233] Waiting for 1 tablets on tserver eb5cc48c49024c098a500461c6595b05 to finish bootstrapping
I20260924 12:48:35.692862 5431 cluster_itest_util.cc:256] Not converged past 1 yet: 0.0 0.0 1.1
I20260924 12:48:35.796159 5431 cluster_itest_util.cc:256] Not converged past 1 yet: 0.0 0.0 1.1
I20260924 12:48:35.999681 5431 cluster_itest_util.cc:256] Not converged past 1 yet: 0.0 0.0 1.1
I20260924 12:48:36.079738 13313 consensus_queue.cc:1035] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "229859ec43934d368d51025f812bb247" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:48:36.087414 13313 consensus_queue.cc:1035] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5cc48c49024c098a500461c6595b05" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 42825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:48:36.317325 5431 test_util.cc:268] Using random seed: 78194783
I20260924 12:48:36.330994 5431 test_workload.cc:401] TestWorkload: Skipping table creation because table TestTable already exists
I20260924 12:48:36.332513 5431 raft_consensus-itest-base.cc:268] Waiting until we've written at least 4MB...
I20260924 12:48:36.405454 13151 maintenance_manager.cc:419] P 229859ec43934d368d51025f812bb247: Scheduling FlushMRSOp(6796266c3d914a69945997c45a5e42fb): 1709343 bytes log retention, and flush 2998537 bytes memory
I20260924 12:48:36.463869 13020 maintenance_manager.cc:419] P 84b1720d1c674777bd8a4d94314a4c77: Scheduling FlushMRSOp(6796266c3d914a69945997c45a5e42fb): 3681746 bytes log retention, and flush 5079305 bytes memory
I20260924 12:48:36.492683 5431 raft_consensus-itest-base.cc:274] Waiting for log GC on 84b1720d1c674777bd8a4d94314a4c77
I20260924 12:48:36.551483 13042 maintenance_manager.cc:643] P 229859ec43934d368d51025f812bb247: FlushMRSOp(6796266c3d914a69945997c45a5e42fb) complete. Timing: real 0.145s user 0.065s sys 0.036s Metrics: {"bytes_written":135803,"cfile_init":1,"compiler_manager_pool.queue_time_us":18685,"dirs.queue_time_us":19953,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":985,"drs_written":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4,"lbm_write_time_us":955,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":16,"thread_start_us":35451,"threads_started":2}
I20260924 12:48:36.553372 13151 maintenance_manager.cc:419] P 229859ec43934d368d51025f812bb247: Scheduling LogGCOp(6796266c3d914a69945997c45a5e42fb): free 1709343 bytes of WAL
I20260924 12:48:36.553882 13042 log_reader.cc:386] T 6796266c3d914a69945997c45a5e42fb: removed 2 log segments from log reader
I20260924 12:48:36.554085 13042 log.cc:1076] T 6796266c3d914a69945997c45a5e42fb P 229859ec43934d368d51025f812bb247: Deleting log segment in path: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/wals/6796266c3d914a69945997c45a5e42fb/wal-000000001 (ops 1-7)
I20260924 12:48:36.554284 13042 log.cc:1076] T 6796266c3d914a69945997c45a5e42fb P 229859ec43934d368d51025f812bb247: Deleting log segment in path: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/wals/6796266c3d914a69945997c45a5e42fb/wal-000000002 (ops 8-14)
I20260924 12:48:36.555176 13042 maintenance_manager.cc:643] P 229859ec43934d368d51025f812bb247: LogGCOp(6796266c3d914a69945997c45a5e42fb) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {}
I20260924 12:48:36.555641 13151 maintenance_manager.cc:419] P 229859ec43934d368d51025f812bb247: Scheduling FlushMRSOp(6796266c3d914a69945997c45a5e42fb): 2761278 bytes log retention, and flush 4038921 bytes memory
I20260924 12:48:36.593247 12911 maintenance_manager.cc:643] P 84b1720d1c674777bd8a4d94314a4c77: FlushMRSOp(6796266c3d914a69945997c45a5e42fb) complete. Timing: real 0.128s user 0.086s sys 0.022s Metrics: {"bytes_written":135937,"cfile_init":1,"compiler_manager_pool.queue_time_us":20395,"dirs.queue_time_us":6465,"dirs.run_cpu_time_us":219,"dirs.run_wall_time_us":5876,"drs_written":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4,"lbm_write_time_us":939,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":29,"thread_start_us":26852,"threads_started":2}
I20260924 12:48:36.595072 13020 maintenance_manager.cc:419] P 84b1720d1c674777bd8a4d94314a4c77: Scheduling LogGCOp(6796266c3d914a69945997c45a5e42fb): free 2761366 bytes of WAL
I20260924 12:48:36.595564 12911 log_reader.cc:386] T 6796266c3d914a69945997c45a5e42fb: removed 3 log segments from log reader
I20260924 12:48:36.595748 12911 log.cc:1076] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77: Deleting log segment in path: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/wals/6796266c3d914a69945997c45a5e42fb/wal-000000001 (ops 1-8)
I20260924 12:48:36.595961 12911 log.cc:1076] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77: Deleting log segment in path: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/wals/6796266c3d914a69945997c45a5e42fb/wal-000000002 (ops 9-15)
I20260924 12:48:36.596103 12911 log.cc:1076] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77: Deleting log segment in path: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/wals/6796266c3d914a69945997c45a5e42fb/wal-000000003 (ops 16-22)
I20260924 12:48:36.597206 12911 maintenance_manager.cc:643] P 84b1720d1c674777bd8a4d94314a4c77: LogGCOp(6796266c3d914a69945997c45a5e42fb) complete. Timing: real 0.002s user 0.001s sys 0.000s Metrics: {}
I20260924 12:48:36.597658 13020 maintenance_manager.cc:419] P 84b1720d1c674777bd8a4d94314a4c77: Scheduling UndoDeltaBlockGCOp(6796266c3d914a69945997c45a5e42fb): 461 bytes on disk
I20260924 12:48:36.598613 12911 maintenance_manager.cc:643] P 84b1720d1c674777bd8a4d94314a4c77: UndoDeltaBlockGCOp(6796266c3d914a69945997c45a5e42fb) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20260924 12:48:36.606357 5431 raft_consensus-itest-base.cc:279] Log GC complete on 84b1720d1c674777bd8a4d94314a4c77
I20260924 12:48:36.623680 13042 maintenance_manager.cc:643] P 229859ec43934d368d51025f812bb247: FlushMRSOp(6796266c3d914a69945997c45a5e42fb) complete. Timing: real 0.068s user 0.062s sys 0.000s Metrics: {"bytes_written":135869,"cfile_init":1,"dirs.queue_time_us":75,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":1410,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":930,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":20}
I20260924 12:48:36.625061 13151 maintenance_manager.cc:419] P 229859ec43934d368d51025f812bb247: Scheduling LogGCOp(6796266c3d914a69945997c45a5e42fb): free 920446 bytes of WAL
I20260924 12:48:36.625393 13042 log_reader.cc:386] T 6796266c3d914a69945997c45a5e42fb: removed 1 log segments from log reader
I20260924 12:48:36.625478 13042 log.cc:1076] T 6796266c3d914a69945997c45a5e42fb P 229859ec43934d368d51025f812bb247: Deleting log segment in path: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/wals/6796266c3d914a69945997c45a5e42fb/wal-000000003 (ops 15-21)
I20260924 12:48:36.625910 13042 maintenance_manager.cc:643] P 229859ec43934d368d51025f812bb247: LogGCOp(6796266c3d914a69945997c45a5e42fb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20260924 12:48:36.626267 13151 maintenance_manager.cc:419] P 229859ec43934d368d51025f812bb247: Scheduling UndoDeltaBlockGCOp(6796266c3d914a69945997c45a5e42fb): 672 bytes on disk
I20260924 12:48:36.627291 13042 maintenance_manager.cc:643] P 229859ec43934d368d51025f812bb247: UndoDeltaBlockGCOp(6796266c3d914a69945997c45a5e42fb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":176,"lbm_reads_lt_1ms":8}
I20260924 12:48:36.627974 13151 maintenance_manager.cc:419] P 229859ec43934d368d51025f812bb247: Scheduling CompactRowSetsOp(6796266c3d914a69945997c45a5e42fb): perf score=1.007639
I20260924 12:48:36.758123 13042 maintenance_manager.cc:643] P 229859ec43934d368d51025f812bb247: CompactRowSetsOp(6796266c3d914a69945997c45a5e42fb) complete. Timing: real 0.130s user 0.105s sys 0.011s Metrics: {"bytes_written":136001,"cfile_cache_hit":10,"cfile_cache_hit_bytes":262392,"cfile_cache_miss":26,"cfile_cache_miss_bytes":264284,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":2105,"dirs.run_cpu_time_us":881,"dirs.run_wall_time_us":4882,"drs_written":1,"lbm_read_time_us":1164,"lbm_reads_lt_1ms":46,"lbm_write_time_us":947,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":208,"rows_written":36,"thread_start_us":2047,"threads_started":3}
I20260924 12:48:36.759166 13151 maintenance_manager.cc:419] P 229859ec43934d368d51025f812bb247: Scheduling UndoDeltaBlockGCOp(6796266c3d914a69945997c45a5e42fb): 536 bytes on disk
I20260924 12:48:36.759763 13042 maintenance_manager.cc:643] P 229859ec43934d368d51025f812bb247: UndoDeltaBlockGCOp(6796266c3d914a69945997c45a5e42fb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20260924 12:48:38.607951 5431 raft_consensus-itest-base.cc:295] Servers converged with original term 1
I20260924 12:48:38.608074 5431 raft_consensus-itest-base.cc:300] Resuming eb5cc48c49024c098a500461c6595b05
I20260924 12:48:38.611459 13353 raft_consensus.cc:482] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 84b1720d1c674777bd8a4d94314a4c77)
I20260924 12:48:38.611723 13353 raft_consensus.cc:504] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229859ec43934d368d51025f812bb247" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39323 } } peers { permanent_uuid: "84b1720d1c674777bd8a4d94314a4c77" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46525 } } peers { permanent_uuid: "eb5cc48c49024c098a500461c6595b05" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 42825 } }
I20260924 12:48:38.613454 13353 leader_election.cc:288] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 229859ec43934d368d51025f812bb247 (127.5.77.194:39323), 84b1720d1c674777bd8a4d94314a4c77 (127.5.77.193:46525)
I20260924 12:48:38.622593 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796266c3d914a69945997c45a5e42fb" candidate_uuid: "eb5cc48c49024c098a500461c6595b05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "229859ec43934d368d51025f812bb247" is_pre_election: true
I20260924 12:48:38.624145 13352 consensus_queue.cc:740] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [LEADER]: The logs necessary to catch up peer eb5cc48c49024c098a500461c6595b05 have been garbage collected. The follower will never be able to catch up (Not found: failed to read ops 3..31: Segment 1 which contained index 3 has been GCed)
I20260924 12:48:38.624907 13360 raft_consensus.cc:998] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77: Attempting to remove follower eb5cc48c49024c098a500461c6595b05 from the Raft config. Reason: The logs necessary to catch up peer eb5cc48c49024c098a500461c6595b05 have been garbage collected. The replica will never be able to catch up
I20260924 12:48:38.626335 13360 consensus_queue.cc:237] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 1.37, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "229859ec43934d368d51025f812bb247" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39323 } } peers { permanent_uuid: "84b1720d1c674777bd8a4d94314a4c77" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46525 } }
I20260924 12:48:38.626487 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796266c3d914a69945997c45a5e42fb" candidate_uuid: "eb5cc48c49024c098a500461c6595b05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "84b1720d1c674777bd8a4d94314a4c77" is_pre_election: true
I20260924 12:48:38.627398 13168 leader_election.cc:302] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb5cc48c49024c098a500461c6595b05; no voters: 229859ec43934d368d51025f812bb247, 84b1720d1c674777bd8a4d94314a4c77
I20260924 12:48:38.628065 13353 raft_consensus.cc:2726] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 12:48:38.628468 13105 raft_consensus.cc:1261] T 6796266c3d914a69945997c45a5e42fb P 229859ec43934d368d51025f812bb247 [term 1 FOLLOWER]: Refusing update from remote peer 84b1720d1c674777bd8a4d94314a4c77: Log matching property violated. Preceding OpId in replica: term: 1 index: 37. Preceding OpId from leader: term: 1 index: 38. (index mismatch)
I20260924 12:48:38.628931 13352 consensus_queue.cc:1035] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "229859ec43934d368d51025f812bb247" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20260924 12:48:38.631438 13360 raft_consensus.cc:2932] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [term 1 LEADER]: Committing config change with OpId 1.38: config changed from index -1 to 38, VOTER eb5cc48c49024c098a500461c6595b05 (127.5.77.195) evicted. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "229859ec43934d368d51025f812bb247" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39323 } } peers { permanent_uuid: "84b1720d1c674777bd8a4d94314a4c77" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46525 } } }
I20260924 12:48:38.631891 13105 raft_consensus.cc:2932] T 6796266c3d914a69945997c45a5e42fb P 229859ec43934d368d51025f812bb247 [term 1 FOLLOWER]: Committing config change with OpId 1.38: config changed from index -1 to 38, VOTER eb5cc48c49024c098a500461c6595b05 (127.5.77.195) evicted. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "229859ec43934d368d51025f812bb247" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39323 } } peers { permanent_uuid: "84b1720d1c674777bd8a4d94314a4c77" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46525 } } }
I20260924 12:48:38.635454 12830 catalog_manager.cc:5495] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 reported cstate change: config changed from index -1 to 38, VOTER eb5cc48c49024c098a500461c6595b05 (127.5.77.195) evicted. New cstate: current_term: 1 leader_uuid: "84b1720d1c674777bd8a4d94314a4c77" committed_config { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "229859ec43934d368d51025f812bb247" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39323 } } peers { permanent_uuid: "84b1720d1c674777bd8a4d94314a4c77" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46525 } } }
I20260924 12:48:38.640781 13215 tablet_service.cc:1514] Processing DeleteTablet for tablet 6796266c3d914a69945997c45a5e42fb with delete_type TABLET_DATA_TOMBSTONED (TS eb5cc48c49024c098a500461c6595b05 not found in new config with opid_index 38) from {username='slave'} at 127.0.0.1:34670
I20260924 12:48:38.641614 13368 tablet_replica.cc:330] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: stopping tablet replica
I20260924 12:48:38.642000 13368 raft_consensus.cc:2229] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:48:38.642343 13368 pending_rounds.cc:62] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: Trying to abort 1 pending ops.
I20260924 12:48:38.642475 13368 pending_rounds.cc:68] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7332880860568281088 op_type: WRITE_OP write_request { tablet_id: "6796266c3d914a69945997c45a5e42fb" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\\w\231}\205Q\363\304\000\000\000\000\000\000\000\000\000\000\002\000\000\000\000\000" indirect_data: "<redacted>""00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
I20260924 12:48:38.644905 13368 raft_consensus.cc:2258] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:48:38.646139 13368 ts_tablet_manager.cc:1905] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 12:48:38.650393 13368 ts_tablet_manager.cc:1918] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20260924 12:48:38.650563 13368 log.cc:1196] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: Deleting WAL directory at /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal/wals/6796266c3d914a69945997c45a5e42fb
I20260924 12:48:38.651376 12818 catalog_manager.cc:4841] TS eb5cc48c49024c098a500461c6595b05 (127.5.77.195:42825): tablet 6796266c3d914a69945997c45a5e42fb (table TestTable [id=a7ee2e5f53fc49ddb5df3764f131bcd8]) successfully deleted
I20260924 12:48:38.658591 13275 generic_service.cc:175] {username='slave'} at 127.0.0.1:34682 changed flags via RPC: tablet_copy_fault_crash_on_fetch_all from '0' to '1.0'
I20260924 12:48:38.660586 12974 consensus_queue.cc:237] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 1.38, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "229859ec43934d368d51025f812bb247" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39323 } } peers { permanent_uuid: "84b1720d1c674777bd8a4d94314a4c77" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46525 } } peers { permanent_uuid: "eb5cc48c49024c098a500461c6595b05" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 42825 } attrs { } }
I20260924 12:48:38.662590 13105 raft_consensus.cc:1261] T 6796266c3d914a69945997c45a5e42fb P 229859ec43934d368d51025f812bb247 [term 1 FOLLOWER]: Refusing update from remote peer 84b1720d1c674777bd8a4d94314a4c77: Log matching property violated. Preceding OpId in replica: term: 1 index: 38. Preceding OpId from leader: term: 1 index: 39. (index mismatch)
W20260924 12:48:38.662973 12908 consensus_peers.cc:487] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 -> Peer eb5cc48c49024c098a500461c6595b05 (127.5.77.195:42825): Couldn't send request to peer eb5cc48c49024c098a500461c6595b05. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260924 12:48:38.663123 13352 consensus_queue.cc:1035] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "229859ec43934d368d51025f812bb247" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20260924 12:48:38.665069 13360 raft_consensus.cc:2932] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [term 1 LEADER]: Committing config change with OpId 1.39: config changed from index 38 to 39, VOTER eb5cc48c49024c098a500461c6595b05 (127.5.77.195) added. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "229859ec43934d368d51025f812bb247" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39323 } } peers { permanent_uuid: "84b1720d1c674777bd8a4d94314a4c77" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46525 } } peers { permanent_uuid: "eb5cc48c49024c098a500461c6595b05" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 42825 } attrs { } } }
I20260924 12:48:38.665607 13105 raft_consensus.cc:2932] T 6796266c3d914a69945997c45a5e42fb P 229859ec43934d368d51025f812bb247 [term 1 FOLLOWER]: Committing config change with OpId 1.39: config changed from index 38 to 39, VOTER eb5cc48c49024c098a500461c6595b05 (127.5.77.195) added. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "229859ec43934d368d51025f812bb247" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39323 } } peers { permanent_uuid: "84b1720d1c674777bd8a4d94314a4c77" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46525 } } peers { permanent_uuid: "eb5cc48c49024c098a500461c6595b05" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 42825 } attrs { } } }
I20260924 12:48:38.669149 12830 catalog_manager.cc:5495] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 reported cstate change: config changed from index 38 to 39, VOTER eb5cc48c49024c098a500461c6595b05 (127.5.77.195) added. New cstate: current_term: 1 leader_uuid: "84b1720d1c674777bd8a4d94314a4c77" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "229859ec43934d368d51025f812bb247" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39323 } } peers { permanent_uuid: "84b1720d1c674777bd8a4d94314a4c77" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46525 } } peers { permanent_uuid: "eb5cc48c49024c098a500461c6595b05" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 42825 } attrs { } } }
I20260924 12:48:39.239140 13370 ts_tablet_manager.cc:927] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: Initiating tablet copy from peer 84b1720d1c674777bd8a4d94314a4c77 (127.5.77.193:46525)
I20260924 12:48:39.240033 13370 tablet_copy_client.cc:316] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: tablet copy: Beginning tablet copy session from remote peer at address 127.5.77.193:46525
I20260924 12:48:39.241007 12994 tablet_copy_service.cc:140] P 84b1720d1c674777bd8a4d94314a4c77: Received BeginTabletCopySession request for tablet 6796266c3d914a69945997c45a5e42fb from peer eb5cc48c49024c098a500461c6595b05 ({username='slave'} at 127.5.77.195:57111)
I20260924 12:48:39.241235 12994 tablet_copy_service.cc:161] P 84b1720d1c674777bd8a4d94314a4c77: Beginning new tablet copy session on tablet 6796266c3d914a69945997c45a5e42fb from peer eb5cc48c49024c098a500461c6595b05 at {username='slave'} at 127.5.77.195:57111: session id = eb5cc48c49024c098a500461c6595b05-6796266c3d914a69945997c45a5e42fb
I20260924 12:48:39.244396 12994 tablet_copy_source_session.cc:205] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77: Tablet Copy: opened 5 blocks and 3 log segments
I20260924 12:48:39.246531 13370 ts_tablet_manager.cc:1905] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260924 12:48:39.250989 13370 ts_tablet_manager.cc:1918] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.2
I20260924 12:48:39.251271 13370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6796266c3d914a69945997c45a5e42fb. 1 dirs total, 0 dirs full, 0 dirs failed
W20260924 12:48:39.253522 13370 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260924 12:48:39.253629 13370 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260924 12:48:39.253695 13370 fault_injection.cc:59] injecting fault for kudu (pid 13153): FLAGS_tablet_copy_fault_crash_on_fetch_all (process will exit)
I20260924 12:48:39.329957 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 12892
I20260924 12:48:39.331278 13014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:39.448115 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 12892
I20260924 12:48:39.466755 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 13023
I20260924 12:48:39.467860 13145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:39.580685 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 13023
I20260924 12:48:39.598567 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 12802
I20260924 12:48:39.599675 12863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:39.703162 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 12802
I20260924 12:48:39.722204 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.77.254:38013
--webserver_interface=127.5.77.254
--webserver_port=43969
--builtin_ntp_servers=127.5.77.212:35701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.77.254:38013
--master_add_server_when_underreplicated=false
--raft_prepare_replacement_before_eviction=false with env {}
W20260924 12:48:40.042413 13375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:40.042765 13375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:40.042887 13375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:40.051002 13375 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260924 12:48:40.051144 13375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:48:40.051206 13375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:40.051282 13375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:48:40.051342 13375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:48:40.061096 13375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:35701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
--master_add_server_when_underreplicated=false
--ipki_ca_key_size=768
--master_addresses=127.5.77.254:38013
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.77.254:38013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.77.254
--webserver_port=43969
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:40.062443 13375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:40.064432 13375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:40.076181 13383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:40.076198 13381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:40.076236 13380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:40.077430 13375 server_base.cc:1034] running on GCE node
I20260924 12:48:40.078291 13375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:40.079870 13375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:40.081080 13375 hybrid_clock.cc:648] HybridClock initialized: now 1790254120081033 us; error 46 us; skew 500 ppm
I20260924 12:48:40.081444 13375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:40.086958 13375 webserver.cc:460] Webserver started at http://127.5.77.254:43969/ using document root <none> and password file <none>
I20260924 12:48:40.087709 13375 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:40.087848 13375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:40.093307 13375 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 12:48:40.096261 13389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:40.097617 13375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 12:48:40.097785 13375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
uuid: "eb6156389fe548ceb292b29fa38d42f2"
format_stamp: "Formatted at 2026-09-24 12:48:33 on dist-test-slave-b8q3"
I20260924 12:48:40.098758 13375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:40.125294 13375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:40.126196 13375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:40.126508 13375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:40.143615 13375 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.254:38013
I20260924 12:48:40.143709 13440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.254:38013 every 8 connection(s)
I20260924 12:48:40.145105 13375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260924 12:48:40.151669 13441 sys_catalog.cc:263] Verifying existing consensus state
I20260924 12:48:40.153544 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 13375
I20260924 12:48:40.154531 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.193:46525
--local_ip_for_outbound_sockets=127.5.77.193
--tserver_master_addrs=127.5.77.254:38013
--webserver_port=32777
--webserver_interface=127.5.77.193
--builtin_ntp_servers=127.5.77.212:35701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20260924 12:48:40.155786 13441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2: Bootstrap starting.
I20260924 12:48:40.173861 13441 log.cc:824] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:40.185406 13441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 12:48:40.185891 13441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2: Bootstrap complete.
I20260924 12:48:40.194648 13441 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6156389fe548ceb292b29fa38d42f2" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 38013 } }
I20260924 12:48:40.195500 13441 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6156389fe548ceb292b29fa38d42f2, State: Initialized, Role: FOLLOWER
I20260924 12:48:40.196162 13441 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6156389fe548ceb292b29fa38d42f2" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 38013 } }
I20260924 12:48:40.196377 13441 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:48:40.196486 13441 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:48:40.196653 13441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:48:40.200206 13441 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6156389fe548ceb292b29fa38d42f2" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 38013 } }
I20260924 12:48:40.200702 13441 leader_election.cc:302] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb6156389fe548ceb292b29fa38d42f2; no voters:
I20260924 12:48:40.201268 13441 leader_election.cc:288] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 12:48:40.201418 13445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 12:48:40.202476 13445 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [term 2 LEADER]: Becoming Leader. State: Replica: eb6156389fe548ceb292b29fa38d42f2, State: Running, Role: LEADER
I20260924 12:48:40.203086 13445 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6156389fe548ceb292b29fa38d42f2" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 38013 } }
I20260924 12:48:40.203462 13441 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:48:40.206665 13447 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb6156389fe548ceb292b29fa38d42f2. Latest consensus state: current_term: 2 leader_uuid: "eb6156389fe548ceb292b29fa38d42f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6156389fe548ceb292b29fa38d42f2" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 38013 } } }
I20260924 12:48:40.207000 13447 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [sys.catalog]: This master's current role is: LEADER
I20260924 12:48:40.206360 13446 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "eb6156389fe548ceb292b29fa38d42f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6156389fe548ceb292b29fa38d42f2" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 38013 } } }
I20260924 12:48:40.207547 13446 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2 [sys.catalog]: This master's current role is: LEADER
I20260924 12:48:40.219442 13452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:48:40.225687 13452 catalog_manager.cc:669] Loaded metadata for table TestTable [id=a7ee2e5f53fc49ddb5df3764f131bcd8]
I20260924 12:48:40.228402 13452 tablet_loader.cc:96] loaded metadata for tablet 6796266c3d914a69945997c45a5e42fb (table TestTable [id=a7ee2e5f53fc49ddb5df3764f131bcd8])
I20260924 12:48:40.228991 13452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:48:40.230597 13452 catalog_manager.cc:1259] Loaded cluster ID: 01d317af27644887acb13ae3348c26c6
I20260924 12:48:40.230785 13452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:48:40.235113 13452 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:48:40.236874 13452 catalog_manager.cc:5879] T 00000000000000000000000000000000 P eb6156389fe548ceb292b29fa38d42f2: Loaded TSK: 0
I20260924 12:48:40.238694 13452 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 12:48:40.512657 13443 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20260924 12:48:40.513015 13443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:40.513099 13443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:40.513212 13443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:40.520658 13443 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20260924 12:48:40.520754 13443 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20260924 12:48:40.520835 13443 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260924 12:48:40.520905 13443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:40.521045 13443 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20260924 12:48:40.521109 13443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.193
I20260924 12:48:40.530378 13443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:35701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.193:46525
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.77.193
--webserver_port=32777
--tserver_master_addrs=127.5.77.254:38013
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.193
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:40.531754 13443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:40.533516 13443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:40.546751 13467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:40.546761 13470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:40.546694 13468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:40.548043 13443 server_base.cc:1034] running on GCE node
I20260924 12:48:40.548820 13443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:40.550063 13443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:40.551321 13443 hybrid_clock.cc:648] HybridClock initialized: now 1790254120551247 us; error 60 us; skew 500 ppm
I20260924 12:48:40.551723 13443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:40.554558 13443 webserver.cc:460] Webserver started at http://127.5.77.193:32777/ using document root <none> and password file <none>
I20260924 12:48:40.555514 13443 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:40.555649 13443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:40.562822 13443 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 12:48:40.572329 13479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:40.573819 13443 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.004s sys 0.000s
I20260924 12:48:40.574008 13443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "84b1720d1c674777bd8a4d94314a4c77"
format_stamp: "Formatted at 2026-09-24 12:48:34 on dist-test-slave-b8q3"
I20260924 12:48:40.575027 13443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 5
Total live bytes: 136398
Total live bytes (after alignment): 155648
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:40.595947 13443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:40.596843 13443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:40.597163 13443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:40.598193 13443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:40.600230 13486 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260924 12:48:40.606338 13443 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 12:48:40.606446 13443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260924 12:48:40.606600 13443 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 12:48:40.609586 13443 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 12:48:40.609691 13443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260924 12:48:40.610086 13486 tablet_bootstrap.cc:492] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77: Bootstrap starting.
I20260924 12:48:40.630530 13486 log.cc:824] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:40.645830 13443 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.193:46525
I20260924 12:48:40.646082 13593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.193:46525 every 8 connection(s)
I20260924 12:48:40.647313 13443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260924 12:48:40.647787 13486 tablet_bootstrap.cc:492] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77: Bootstrap replayed 1/3 log segments. Stats: ops{read=7 overwritten=0 applied=4 ignored=4} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=1. Pending: 3 replicates
I20260924 12:48:40.651165 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 13443
I20260924 12:48:40.651934 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.195:42825
--local_ip_for_outbound_sockets=127.5.77.195
--tserver_master_addrs=127.5.77.254:38013
--webserver_port=36971
--webserver_interface=127.5.77.195
--builtin_ntp_servers=127.5.77.212:35701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20260924 12:48:40.664937 13486 tablet_bootstrap.cc:492] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77: Bootstrap replayed 2/3 log segments. Stats: ops{read=14 overwritten=0 applied=12 ignored=8} inserts{seen=4 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=1. Pending: 2 replicates
I20260924 12:48:40.672052 13486 tablet_bootstrap.cc:492] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77: Bootstrap replayed 3/3 log segments. Stats: ops{read=17 overwritten=0 applied=17 ignored=8} inserts{seen=7 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=1. Pending: 0 replicates
I20260924 12:48:40.672689 13486 tablet_bootstrap.cc:492] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77: Bootstrap complete.
I20260924 12:48:40.673506 13486 ts_tablet_manager.cc:1397] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77: Time spent bootstrapping tablet: real 0.064s user 0.039s sys 0.020s
I20260924 12:48:40.679710 13486 raft_consensus.cc:348] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "229859ec43934d368d51025f812bb247" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39323 } } peers { permanent_uuid: "84b1720d1c674777bd8a4d94314a4c77" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46525 } } peers { permanent_uuid: "eb5cc48c49024c098a500461c6595b05" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 42825 } attrs { } }
I20260924 12:48:40.680780 13486 raft_consensus.cc:729] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84b1720d1c674777bd8a4d94314a4c77, State: Initialized, Role: FOLLOWER
I20260924 12:48:40.681523 13486 consensus_queue.cc:260] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 39, Last appended: 1.39, Last appended by leader: 39, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "229859ec43934d368d51025f812bb247" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39323 } } peers { permanent_uuid: "84b1720d1c674777bd8a4d94314a4c77" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46525 } } peers { permanent_uuid: "eb5cc48c49024c098a500461c6595b05" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 42825 } attrs { } }
I20260924 12:48:40.683367 13486 ts_tablet_manager.cc:1428] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77: Time spent starting tablet: real 0.010s user 0.011s sys 0.001s
I20260924 12:48:40.688387 13594 heartbeater.cc:347] Connected to a master server at 127.5.77.254:38013
I20260924 12:48:40.688663 13594 heartbeater.cc:464] Registering TS with master...
I20260924 12:48:40.689316 13594 heartbeater.cc:511] Master 127.5.77.254:38013 requested a full tablet report, sending...
I20260924 12:48:40.692160 13406 ts_manager.cc:194] Registered new tserver with Master: 84b1720d1c674777bd8a4d94314a4c77 (127.5.77.193:46525)
I20260924 12:48:40.694633 13406 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.193:52575
I20260924 12:48:40.697999 13594 heartbeater.cc:503] Master 127.5.77.254:38013 was elected leader, sending a full tablet report...
I20260924 12:48:40.750017 13595 maintenance_manager.cc:419] P 84b1720d1c674777bd8a4d94314a4c77: Scheduling UndoDeltaBlockGCOp(6796266c3d914a69945997c45a5e42fb): 461 bytes on disk
I20260924 12:48:40.751291 13484 maintenance_manager.cc:643] P 84b1720d1c674777bd8a4d94314a4c77: UndoDeltaBlockGCOp(6796266c3d914a69945997c45a5e42fb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
W20260924 12:48:40.984112 13598 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20260924 12:48:40.984493 13598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:40.984577 13598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:40.984690 13598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:40.991940 13598 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20260924 12:48:40.992039 13598 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20260924 12:48:40.992132 13598 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260924 12:48:40.992208 13598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:40.992367 13598 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20260924 12:48:40.992431 13598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.195
I20260924 12:48:41.001624 13598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:35701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.195:42825
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.77.195
--webserver_port=36971
--tserver_master_addrs=127.5.77.254:38013
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.195
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:41.002921 13598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:41.004742 13598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:41.017077 13604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:41.017079 13607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:41.017040 13605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:41.017607 13598 server_base.cc:1034] running on GCE node
I20260924 12:48:41.018239 13598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:41.019244 13598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:41.020447 13598 hybrid_clock.cc:648] HybridClock initialized: now 1790254121020379 us; error 59 us; skew 500 ppm
I20260924 12:48:41.020757 13598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:41.022886 13598 webserver.cc:460] Webserver started at http://127.5.77.195:36971/ using document root <none> and password file <none>
I20260924 12:48:41.023558 13598 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:41.023667 13598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:41.028856 13598 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 12:48:41.031505 13613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:41.032790 13598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 12:48:41.032934 13598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "eb5cc48c49024c098a500461c6595b05"
format_stamp: "Formatted at 2026-09-24 12:48:35 on dist-test-slave-b8q3"
I20260924 12:48:41.033828 13598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:41.059787 13598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:41.060609 13598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:41.060892 13598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:41.061856 13598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:41.063752 13620 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260924 12:48:41.068813 13620 ts_tablet_manager.cc:1852] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_COPYING
I20260924 12:48:41.072028 13620 ts_tablet_manager.cc:1905] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 12:48:41.078873 13620 ts_tablet_manager.cc:1918] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20260924 12:48:41.079978 13598 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 12:48:41.080082 13598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.017s user 0.001s sys 0.000s
I20260924 12:48:41.080243 13598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:48:41.080296 13598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:41.109828 13598 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.195:42825
I20260924 12:48:41.109851 13726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.195:42825 every 8 connection(s)
I20260924 12:48:41.111198 13598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260924 12:48:41.120326 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 13598
I20260924 12:48:41.120582 5431 test_util.cc:268] Using random seed: 82998071
I20260924 12:48:41.125484 13727 heartbeater.cc:347] Connected to a master server at 127.5.77.254:38013
I20260924 12:48:41.125811 13727 heartbeater.cc:464] Registering TS with master...
I20260924 12:48:41.126516 13727 heartbeater.cc:511] Master 127.5.77.254:38013 requested a full tablet report, sending...
I20260924 12:48:41.128314 13406 ts_manager.cc:194] Registered new tserver with Master: eb5cc48c49024c098a500461c6595b05 (127.5.77.195:42825)
I20260924 12:48:41.129477 13406 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.195:51141
I20260924 12:48:41.136974 5431 test_workload.cc:401] TestWorkload: Skipping table creation because table TestTable already exists
W20260924 12:48:41.184965 13507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.184926 13505 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.184891 13508 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.184999 13506 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.189019 13733 meta_cache.cc:302] tablet 6796266c3d914a69945997c45a5e42fb: replica 229859ec43934d368d51025f812bb247 (127.5.77.194:39323) has failed: Network error: Client connection negotiation failed: client connection to 127.5.77.194:39323: connect: Connection refused (error 111)
W20260924 12:48:41.204789 13506 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.204975 13507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.210688 13506 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.221794 13506 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.222024 13507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.234578 13507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.235529 13507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.248677 13507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.248872 13506 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.249832 13506 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.264173 13506 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.264395 13507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.276188 13507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.287309 13507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.304085 13507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.305011 13507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.323108 13507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.323170 13508 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.323165 13506 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.340520 13508 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.358680 13508 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.376760 13508 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.377709 13508 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.399261 13508 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.399735 13506 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.400045 13507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.422633 13507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.443320 13507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.462502 13507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.486176 13507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.502311 13507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.516489 13507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.538117 13507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.560583 13507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.588419 13507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.588539 13506 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.619057 13506 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.620018 13506 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.650596 13506 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.651177 13507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.679845 13507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.686885 13507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.717988 13507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.748898 13507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.778334 13507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.815274 13507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.824116 13507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.857698 13507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.893492 13506 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.893689 13507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.931704 13507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.931892 13506 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
W20260924 12:48:41.932437 13506 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
I20260924 12:48:41.951251 13742 raft_consensus.cc:482] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:48:41.951519 13742 raft_consensus.cc:504] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "229859ec43934d368d51025f812bb247" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39323 } } peers { permanent_uuid: "84b1720d1c674777bd8a4d94314a4c77" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46525 } } peers { permanent_uuid: "eb5cc48c49024c098a500461c6595b05" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 42825 } attrs { } }
I20260924 12:48:41.953032 13742 leader_election.cc:288] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 229859ec43934d368d51025f812bb247 (127.5.77.194:39323), eb5cc48c49024c098a500461c6595b05 (127.5.77.195:42825)
W20260924 12:48:41.954182 13483 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.77.194:39323: connect: Connection refused (error 111)
W20260924 12:48:41.955660 13483 leader_election.cc:334] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 229859ec43934d368d51025f812bb247 (127.5.77.194:39323): Network error: Client connection negotiation failed: client connection to 127.5.77.194:39323: connect: Connection refused (error 111)
I20260924 12:48:41.960403 13682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796266c3d914a69945997c45a5e42fb" candidate_uuid: "84b1720d1c674777bd8a4d94314a4c77" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "eb5cc48c49024c098a500461c6595b05" is_pre_election: true
I20260924 12:48:41.963310 13682 raft_consensus.cc:1789] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05 [term 1 FOLLOWER]: voting while tombstoned based on last-logged opid 1.2
I20260924 12:48:41.964154 13682 raft_consensus.cc:2454] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84b1720d1c674777bd8a4d94314a4c77 in term 1.
I20260924 12:48:41.964942 13481 leader_election.cc:302] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 84b1720d1c674777bd8a4d94314a4c77, eb5cc48c49024c098a500461c6595b05; no voters: 229859ec43934d368d51025f812bb247
I20260924 12:48:41.965425 13742 raft_consensus.cc:2781] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 12:48:41.965560 13742 raft_consensus.cc:482] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:48:41.965652 13742 raft_consensus.cc:3037] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:48:41.969310 13742 raft_consensus.cc:504] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "229859ec43934d368d51025f812bb247" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39323 } } peers { permanent_uuid: "84b1720d1c674777bd8a4d94314a4c77" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46525 } } peers { permanent_uuid: "eb5cc48c49024c098a500461c6595b05" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 42825 } attrs { } }
I20260924 12:48:41.969908 13742 leader_election.cc:288] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [CANDIDATE]: Term 2 election: Requested vote from peers 229859ec43934d368d51025f812bb247 (127.5.77.194:39323), eb5cc48c49024c098a500461c6595b05 (127.5.77.195:42825)
I20260924 12:48:41.970439 13682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796266c3d914a69945997c45a5e42fb" candidate_uuid: "84b1720d1c674777bd8a4d94314a4c77" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "eb5cc48c49024c098a500461c6595b05"
I20260924 12:48:41.970757 13682 raft_consensus.cc:1789] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05 [term 1 FOLLOWER]: voting while tombstoned based on last-logged opid 1.2
I20260924 12:48:41.970932 13682 raft_consensus.cc:3037] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05 [term 1 FOLLOWER]: Advancing to term 2
W20260924 12:48:41.971422 13483 leader_election.cc:334] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 229859ec43934d368d51025f812bb247 (127.5.77.194:39323): Network error: Client connection negotiation failed: client connection to 127.5.77.194:39323: connect: Connection refused (error 111)
W20260924 12:48:41.973106 13506 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
I20260924 12:48:41.973735 13682 raft_consensus.cc:2454] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84b1720d1c674777bd8a4d94314a4c77 in term 2.
I20260924 12:48:41.974121 13481 leader_election.cc:302] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 84b1720d1c674777bd8a4d94314a4c77, eb5cc48c49024c098a500461c6595b05; no voters: 229859ec43934d368d51025f812bb247
W20260924 12:48:41.974373 13506 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51806: Illegal state: replica 84b1720d1c674777bd8a4d94314a4c77 is not leader of this config: current role FOLLOWER
I20260924 12:48:41.974418 13742 raft_consensus.cc:2781] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 12:48:41.974887 13742 raft_consensus.cc:686] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [term 2 LEADER]: Becoming Leader. State: Replica: 84b1720d1c674777bd8a4d94314a4c77, State: Running, Role: LEADER
I20260924 12:48:41.975450 13742 consensus_queue.cc:237] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 1.39, Last appended by leader: 39, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "229859ec43934d368d51025f812bb247" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39323 } } peers { permanent_uuid: "84b1720d1c674777bd8a4d94314a4c77" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46525 } } peers { permanent_uuid: "eb5cc48c49024c098a500461c6595b05" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 42825 } attrs { } }
I20260924 12:48:41.978938 13405 catalog_manager.cc:5495] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "84b1720d1c674777bd8a4d94314a4c77" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "229859ec43934d368d51025f812bb247" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39323 } } peers { permanent_uuid: "84b1720d1c674777bd8a4d94314a4c77" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46525 } } peers { permanent_uuid: "eb5cc48c49024c098a500461c6595b05" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 42825 } attrs { } } }
W20260924 12:48:42.017616 13481 consensus_peers.cc:487] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 -> Peer eb5cc48c49024c098a500461c6595b05 (127.5.77.195:42825): Couldn't send request to peer eb5cc48c49024c098a500461c6595b05. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260924 12:48:42.017984 13483 consensus_peers.cc:487] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77 -> Peer 229859ec43934d368d51025f812bb247 (127.5.77.194:39323): Couldn't send request to peer 229859ec43934d368d51025f812bb247. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.194:39323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 12:48:42.132361 13727 heartbeater.cc:503] Master 127.5.77.254:38013 was elected leader, sending a full tablet report...
I20260924 12:48:42.462064 13754 tablet_replica.cc:330] stopping tablet replica
I20260924 12:48:42.462316 13754 raft_consensus.cc:2229] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260924 12:48:42.462509 13754 raft_consensus.cc:2258] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260924 12:48:42.462821 13754 ts_tablet_manager.cc:927] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: Initiating tablet copy from peer 84b1720d1c674777bd8a4d94314a4c77 (127.5.77.193:46525)
I20260924 12:48:42.463657 13754 tablet_copy_client.cc:316] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: tablet copy: Beginning tablet copy session from remote peer at address 127.5.77.193:46525
I20260924 12:48:42.470300 13569 tablet_copy_service.cc:140] P 84b1720d1c674777bd8a4d94314a4c77: Received BeginTabletCopySession request for tablet 6796266c3d914a69945997c45a5e42fb from peer eb5cc48c49024c098a500461c6595b05 ({username='slave'} at 127.5.77.195:37007)
I20260924 12:48:42.470562 13569 tablet_copy_service.cc:161] P 84b1720d1c674777bd8a4d94314a4c77: Beginning new tablet copy session on tablet 6796266c3d914a69945997c45a5e42fb from peer eb5cc48c49024c098a500461c6595b05 at {username='slave'} at 127.5.77.195:37007: session id = eb5cc48c49024c098a500461c6595b05-6796266c3d914a69945997c45a5e42fb
I20260924 12:48:42.473680 13569 tablet_copy_source_session.cc:205] T 6796266c3d914a69945997c45a5e42fb P 84b1720d1c674777bd8a4d94314a4c77: Tablet Copy: opened 5 blocks and 3 log segments
I20260924 12:48:42.476274 13754 ts_tablet_manager.cc:1905] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260924 12:48:42.480562 13754 ts_tablet_manager.cc:1918] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.2
I20260924 12:48:42.481024 13754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6796266c3d914a69945997c45a5e42fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:42.483507 13754 tablet_copy_client.cc:791] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: tablet copy: Starting download of 5 data blocks...
I20260924 12:48:42.493132 13754 tablet_copy_client.cc:655] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: tablet copy: Starting download of 3 WAL segments...
I20260924 12:48:42.502327 13754 tablet_copy_client.cc:531] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 12:48:42.505751 13754 tablet_bootstrap.cc:492] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: Bootstrap starting.
I20260924 12:48:42.516371 13754 log.cc:824] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:42.527321 13754 tablet_bootstrap.cc:492] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: Bootstrap replayed 1/3 log segments. Stats: ops{read=7 overwritten=0 applied=4 ignored=4} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20260924 12:48:42.539054 13754 tablet_bootstrap.cc:492] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: Bootstrap replayed 2/3 log segments. Stats: ops{read=14 overwritten=0 applied=12 ignored=8} inserts{seen=4 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20260924 12:48:42.545526 13754 tablet_bootstrap.cc:492] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: Bootstrap replayed 3/3 log segments. Stats: ops{read=22 overwritten=0 applied=17 ignored=8} inserts{seen=7 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 5 replicates
I20260924 12:48:42.546089 13754 tablet_bootstrap.cc:492] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: Bootstrap complete.
I20260924 12:48:42.546847 13754 ts_tablet_manager.cc:1397] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: Time spent bootstrapping tablet: real 0.041s user 0.025s sys 0.018s
I20260924 12:48:42.549707 13754 raft_consensus.cc:348] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05 [term 2 FOLLOWER]: Replica starting. Triggering 5 pending ops. Active config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "229859ec43934d368d51025f812bb247" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39323 } } peers { permanent_uuid: "84b1720d1c674777bd8a4d94314a4c77" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46525 } } peers { permanent_uuid: "eb5cc48c49024c098a500461c6595b05" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 42825 } attrs { } }
I20260924 12:48:42.552554 13754 raft_consensus.cc:729] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb5cc48c49024c098a500461c6595b05, State: Initialized, Role: FOLLOWER
I20260924 12:48:42.553174 13754 consensus_queue.cc:260] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 39, Last appended: 2.44, Last appended by leader: 44, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "229859ec43934d368d51025f812bb247" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39323 } } peers { permanent_uuid: "84b1720d1c674777bd8a4d94314a4c77" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 46525 } } peers { permanent_uuid: "eb5cc48c49024c098a500461c6595b05" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 42825 } attrs { } }
I20260924 12:48:42.554689 13754 ts_tablet_manager.cc:1428] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05: Time spent starting tablet: real 0.008s user 0.006s sys 0.002s
I20260924 12:48:42.555691 13568 tablet_copy_service.cc:342] P 84b1720d1c674777bd8a4d94314a4c77: Request end of tablet copy session eb5cc48c49024c098a500461c6595b05-6796266c3d914a69945997c45a5e42fb received from {username='slave'} at 127.5.77.195:37007
I20260924 12:48:42.555822 13568 tablet_copy_service.cc:433] P 84b1720d1c674777bd8a4d94314a4c77: ending tablet copy session eb5cc48c49024c098a500461c6595b05-6796266c3d914a69945997c45a5e42fb on tablet 6796266c3d914a69945997c45a5e42fb with peer eb5cc48c49024c098a500461c6595b05
I20260924 12:48:42.616259 13728 maintenance_manager.cc:419] P eb5cc48c49024c098a500461c6595b05: Scheduling UndoDeltaBlockGCOp(6796266c3d914a69945997c45a5e42fb): 461 bytes on disk
I20260924 12:48:42.617584 13618 maintenance_manager.cc:643] P eb5cc48c49024c098a500461c6595b05: UndoDeltaBlockGCOp(6796266c3d914a69945997c45a5e42fb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20260924 12:48:43.029150 13682 raft_consensus.cc:1203] T 6796266c3d914a69945997c45a5e42fb P eb5cc48c49024c098a500461c6595b05 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.39->[2.40-2.44] Dedup: 2.44->[]
I20260924 12:48:43.035907 13767 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:51806 (ReqId={client: 1392ab48f0754ab8b78b3b2f953b9e77, seq_no=1, attempt_no=36}) took 1022 ms. Trace:
I20260924 12:48:43.036188 13767 rpcz_store.cc:276] 0924 12:48:42.013006 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:48:42.013074 (+ 68us) service_pool.cc:227] Handling call
0924 12:48:43.035836 (+1022762us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:48:42.014027 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6796266c3d914a69945997c45a5e42fb
0924 12:48:42.014188 (+ 161us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:48:42.014193 (+ 5us) write_op.cc:435] Acquired schema lock
0924 12:48:42.014194 (+ 1us) tablet.cc:660] Decoding operations
0924 12:48:42.014720 (+ 526us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:48:42.014733 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0924 12:48:42.014737 (+ 4us) tablet.cc:683] Acquiring locks for 50 operations
0924 12:48:42.015126 (+ 389us) tablet.cc:699] Row locks acquired
0924 12:48:42.015129 (+ 3us) write_op.cc:260] PREPARE: finished
0924 12:48:42.015219 (+ 90us) write_op.cc:270] Start()
0924 12:48:42.015279 (+ 60us) write_op.cc:276] Timestamp: P: 1790254122015151 usec, L: 0
0924 12:48:42.015285 (+ 6us) op_driver.cc:331] REPLICATION: starting
0924 12:48:42.015566 (+ 281us) log.cc:842] Serialized 2455 byte log entry
0924 12:48:43.031368 (+1015802us) op_driver.cc:444] REPLICATION: finished
0924 12:48:43.031850 (+ 482us) write_op.cc:301] APPLY: starting
0924 12:48:43.031875 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
0924 12:48:43.033785 (+ 1910us) tablet.cc:1368] finished BulkCheckPresence
0924 12:48:43.033839 (+ 54us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:48:43.034961 (+ 1122us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:48:43.035154 (+ 193us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=47,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:48:43.035162 (+ 8us) write_op.cc:312] APPLY: finished
0924 12:48:43.035342 (+ 180us) log.cc:842] Serialized 319 byte log entry
0924 12:48:43.035692 (+ 350us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:48:43.035745 (+ 53us) write_op.cc:454] Released schema lock
0924 12:48:43.035756 (+ 11us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":384,"cfile_cache_miss":2,"cfile_cache_miss_bytes":4127,"cfile_init":1,"lbm_read_time_us":147,"lbm_reads_lt_1ms":6,"num_ops":50,"prepare.queue_time_us":465,"prepare.run_cpu_time_us":1772,"prepare.run_wall_time_us":2304,"raft.queue_time_us":88,"raft.run_cpu_time_us":622,"raft.run_wall_time_us":624,"replication_time_us":1016037,"spinlock_wait_cycles":39168,"thread_start_us":597,"threads_started":2}]]}
I20260924 12:48:43.175968 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 13443
I20260924 12:48:43.183911 13589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:43.277801 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 13443
I20260924 12:48:43.293248 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 13598
I20260924 12:48:43.300482 13722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:43.392742 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 13598
I20260924 12:48:43.407074 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 13375
I20260924 12:48:43.413153 13436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:43.514230 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 13375
2026-09-24T12:48:43Z chronyd exiting
[ OK ] RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy (10298 ms)
[ RUN ] RaftConsensusElectionITest.DisruptiveServerAndSlowWAL
I20260924 12:48:43.553412 5431 test_util.cc:268] Using random seed: 85430888
I20260924 12:48:43.555265 5431 ts_itest-base.cc:112] Starting cluster with:
I20260924 12:48:43.555334 5431 ts_itest-base.cc:113] --------------
I20260924 12:48:43.555402 5431 ts_itest-base.cc:114] 3 tablet servers
I20260924 12:48:43.555464 5431 ts_itest-base.cc:115] 3 replicas per TS
I20260924 12:48:43.555540 5431 ts_itest-base.cc:116] --------------
2026-09-24T12:48:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:48:43Z Disabled control of system clock
I20260924 12:48:43.589012 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.77.254:44385
--webserver_interface=127.5.77.254
--webserver_port=0
--builtin_ntp_servers=127.5.77.212:42361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.77.254:44385 with env {}
W20260924 12:48:43.905553 13791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:43.905865 13791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:43.905951 13791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:43.914364 13791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:48:43.914531 13791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:43.914600 13791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:48:43.914664 13791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:48:43.923897 13791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:42361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.77.254:44385
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.77.254:44385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.77.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:43.925175 13791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:43.926921 13791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:43.938356 13796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:43.938418 13799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:43.938400 13797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:43.939857 13791 server_base.cc:1034] running on GCE node
I20260924 12:48:43.940650 13791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:43.942041 13791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:43.943259 13791 hybrid_clock.cc:648] HybridClock initialized: now 1790254123943184 us; error 45 us; skew 500 ppm
I20260924 12:48:43.943640 13791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:43.945816 13791 webserver.cc:460] Webserver started at http://127.5.77.254:34237/ using document root <none> and password file <none>
I20260924 12:48:43.946542 13791 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:43.946660 13791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:43.947002 13791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:43.949636 13791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "311ee9c4541444bbbf499bd05d1e5445"
format_stamp: "Formatted at 2026-09-24 12:48:43 on dist-test-slave-b8q3"
I20260924 12:48:43.950315 13791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "311ee9c4541444bbbf499bd05d1e5445"
format_stamp: "Formatted at 2026-09-24 12:48:43 on dist-test-slave-b8q3"
I20260924 12:48:43.955354 13791 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 12:48:43.958735 13805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:43.960160 13791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 12:48:43.960304 13791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
uuid: "311ee9c4541444bbbf499bd05d1e5445"
format_stamp: "Formatted at 2026-09-24 12:48:43 on dist-test-slave-b8q3"
I20260924 12:48:43.960490 13791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:43.974170 13791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:43.975025 13791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:43.975303 13791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:43.992144 13791 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.254:44385
I20260924 12:48:43.992177 13856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.254:44385 every 8 connection(s)
I20260924 12:48:43.993482 13791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260924 12:48:43.997234 13857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:44.000662 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 13791
I20260924 12:48:44.000856 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260924 12:48:44.009720 13857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 311ee9c4541444bbbf499bd05d1e5445: Bootstrap starting.
I20260924 12:48:44.012856 13857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 311ee9c4541444bbbf499bd05d1e5445: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:44.013968 13857 log.cc:824] T 00000000000000000000000000000000 P 311ee9c4541444bbbf499bd05d1e5445: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:44.016860 13857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 311ee9c4541444bbbf499bd05d1e5445: No bootstrap required, opened a new log
I20260924 12:48:44.024027 13857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 311ee9c4541444bbbf499bd05d1e5445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311ee9c4541444bbbf499bd05d1e5445" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 44385 } }
I20260924 12:48:44.024405 13857 raft_consensus.cc:374] T 00000000000000000000000000000000 P 311ee9c4541444bbbf499bd05d1e5445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:48:44.024511 13857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 311ee9c4541444bbbf499bd05d1e5445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311ee9c4541444bbbf499bd05d1e5445, State: Initialized, Role: FOLLOWER
I20260924 12:48:44.025146 13857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 311ee9c4541444bbbf499bd05d1e5445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311ee9c4541444bbbf499bd05d1e5445" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 44385 } }
I20260924 12:48:44.025364 13857 raft_consensus.cc:388] T 00000000000000000000000000000000 P 311ee9c4541444bbbf499bd05d1e5445 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:48:44.025486 13857 raft_consensus.cc:482] T 00000000000000000000000000000000 P 311ee9c4541444bbbf499bd05d1e5445 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:48:44.025621 13857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 311ee9c4541444bbbf499bd05d1e5445 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:44.027618 13857 raft_consensus.cc:504] T 00000000000000000000000000000000 P 311ee9c4541444bbbf499bd05d1e5445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311ee9c4541444bbbf499bd05d1e5445" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 44385 } }
I20260924 12:48:44.028128 13857 leader_election.cc:302] T 00000000000000000000000000000000 P 311ee9c4541444bbbf499bd05d1e5445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 311ee9c4541444bbbf499bd05d1e5445; no voters:
I20260924 12:48:44.028723 13857 leader_election.cc:288] T 00000000000000000000000000000000 P 311ee9c4541444bbbf499bd05d1e5445 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:48:44.028891 13862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 311ee9c4541444bbbf499bd05d1e5445 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:44.029508 13862 raft_consensus.cc:686] T 00000000000000000000000000000000 P 311ee9c4541444bbbf499bd05d1e5445 [term 1 LEADER]: Becoming Leader. State: Replica: 311ee9c4541444bbbf499bd05d1e5445, State: Running, Role: LEADER
I20260924 12:48:44.030009 13862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 311ee9c4541444bbbf499bd05d1e5445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311ee9c4541444bbbf499bd05d1e5445" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 44385 } }
I20260924 12:48:44.030809 13857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 311ee9c4541444bbbf499bd05d1e5445 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:48:44.032605 13864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 311ee9c4541444bbbf499bd05d1e5445 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 311ee9c4541444bbbf499bd05d1e5445. Latest consensus state: current_term: 1 leader_uuid: "311ee9c4541444bbbf499bd05d1e5445" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311ee9c4541444bbbf499bd05d1e5445" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 44385 } } }
I20260924 12:48:44.033272 13864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 311ee9c4541444bbbf499bd05d1e5445 [sys.catalog]: This master's current role is: LEADER
I20260924 12:48:44.033169 13863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 311ee9c4541444bbbf499bd05d1e5445 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "311ee9c4541444bbbf499bd05d1e5445" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311ee9c4541444bbbf499bd05d1e5445" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 44385 } } }
I20260924 12:48:44.034597 13863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 311ee9c4541444bbbf499bd05d1e5445 [sys.catalog]: This master's current role is: LEADER
I20260924 12:48:44.034860 13870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:48:44.039103 13870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:48:44.047493 13870 catalog_manager.cc:1347] Generated new cluster ID: 4daa3b5bba874aa7a07bcb8023240da1
I20260924 12:48:44.047611 13870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:48:44.080813 13870 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:48:44.082182 13870 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:48:44.087895 13870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 311ee9c4541444bbbf499bd05d1e5445: Generated new TSK 0
I20260924 12:48:44.088758 13870 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:48:44.106700 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.193:0
--local_ip_for_outbound_sockets=127.5.77.193
--webserver_interface=127.5.77.193
--webserver_port=0
--tserver_master_addrs=127.5.77.254:44385
--builtin_ntp_servers=127.5.77.212:42361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--leader_failure_max_missed_heartbeat_periods=3 with env {}
W20260924 12:48:44.426978 13881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:44.427256 13881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:44.427333 13881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:44.434653 13881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:44.434823 13881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.193
I20260924 12:48:44.443918 13881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:42361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.77.193
--webserver_port=0
--tserver_master_addrs=127.5.77.254:44385
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.193
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:44.445259 13881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:44.447126 13881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:44.460273 13887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:44.460234 13889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:44.460561 13886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:44.461229 13881 server_base.cc:1034] running on GCE node
I20260924 12:48:44.461831 13881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:44.462802 13881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:44.464028 13881 hybrid_clock.cc:648] HybridClock initialized: now 1790254124463946 us; error 45 us; skew 500 ppm
I20260924 12:48:44.464404 13881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:44.466727 13881 webserver.cc:460] Webserver started at http://127.5.77.193:45817/ using document root <none> and password file <none>
I20260924 12:48:44.467396 13881 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:44.467495 13881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:44.467759 13881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:44.470327 13881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "088892440d344867b94ec0ad3e8346a8"
format_stamp: "Formatted at 2026-09-24 12:48:44 on dist-test-slave-b8q3"
I20260924 12:48:44.471086 13881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "088892440d344867b94ec0ad3e8346a8"
format_stamp: "Formatted at 2026-09-24 12:48:44 on dist-test-slave-b8q3"
I20260924 12:48:44.476070 13881 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 12:48:44.479537 13895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:44.480891 13881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 12:48:44.481043 13881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "088892440d344867b94ec0ad3e8346a8"
format_stamp: "Formatted at 2026-09-24 12:48:44 on dist-test-slave-b8q3"
I20260924 12:48:44.481228 13881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:44.506970 13881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:44.507797 13881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:44.508074 13881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:44.509011 13881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:44.510452 13881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:48:44.510567 13881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:44.510689 13881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:48:44.510740 13881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:44.544214 13881 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.193:37165
I20260924 12:48:44.544298 14007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.193:37165 every 8 connection(s)
I20260924 12:48:44.545543 13881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260924 12:48:44.561348 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 13881
I20260924 12:48:44.561554 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260924 12:48:44.565304 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.194:0
--local_ip_for_outbound_sockets=127.5.77.194
--webserver_interface=127.5.77.194
--webserver_port=0
--tserver_master_addrs=127.5.77.254:44385
--builtin_ntp_servers=127.5.77.212:42361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--leader_failure_max_missed_heartbeat_periods=3 with env {}
I20260924 12:48:44.568365 14008 heartbeater.cc:347] Connected to a master server at 127.5.77.254:44385
I20260924 12:48:44.568727 14008 heartbeater.cc:464] Registering TS with master...
I20260924 12:48:44.569645 14008 heartbeater.cc:511] Master 127.5.77.254:44385 requested a full tablet report, sending...
I20260924 12:48:44.572063 13822 ts_manager.cc:194] Registered new tserver with Master: 088892440d344867b94ec0ad3e8346a8 (127.5.77.193:37165)
I20260924 12:48:44.573549 13822 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.193:49857
W20260924 12:48:44.896509 14012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:44.896801 14012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:44.896881 14012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:44.904136 14012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:44.904332 14012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.194
I20260924 12:48:44.913349 14012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:42361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.77.194
--webserver_port=0
--tserver_master_addrs=127.5.77.254:44385
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.194
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:44.914621 14012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:44.916307 14012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:44.928694 14017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:44.928798 14020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:44.928653 14018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:44.929364 14012 server_base.cc:1034] running on GCE node
I20260924 12:48:44.930004 14012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:44.931017 14012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:44.932251 14012 hybrid_clock.cc:648] HybridClock initialized: now 1790254124932161 us; error 47 us; skew 500 ppm
I20260924 12:48:44.932562 14012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:44.934880 14012 webserver.cc:460] Webserver started at http://127.5.77.194:43999/ using document root <none> and password file <none>
I20260924 12:48:44.935724 14012 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:44.935839 14012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:44.936148 14012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:44.938807 14012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "52fbcc075f81452e8ebea5dfa19c9aa7"
format_stamp: "Formatted at 2026-09-24 12:48:44 on dist-test-slave-b8q3"
I20260924 12:48:44.939535 14012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "52fbcc075f81452e8ebea5dfa19c9aa7"
format_stamp: "Formatted at 2026-09-24 12:48:44 on dist-test-slave-b8q3"
I20260924 12:48:44.944559 14012 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 12:48:44.947966 14026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:44.949347 14012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 12:48:44.949494 14012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "52fbcc075f81452e8ebea5dfa19c9aa7"
format_stamp: "Formatted at 2026-09-24 12:48:44 on dist-test-slave-b8q3"
I20260924 12:48:44.949661 14012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:44.961603 14012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:44.962407 14012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:44.962704 14012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:44.963665 14012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:44.965137 14012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:48:44.965256 14012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:44.965379 14012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:48:44.965431 14012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:44.996397 14012 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.194:39559
I20260924 12:48:44.996414 14138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.194:39559 every 8 connection(s)
I20260924 12:48:44.998131 14012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260924 12:48:45.001102 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 14012
I20260924 12:48:45.001240 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260924 12:48:45.004778 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.195:0
--local_ip_for_outbound_sockets=127.5.77.195
--webserver_interface=127.5.77.195
--webserver_port=0
--tserver_master_addrs=127.5.77.254:44385
--builtin_ntp_servers=127.5.77.212:42361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--leader_failure_max_missed_heartbeat_periods=3 with env {}
I20260924 12:48:45.012360 14139 heartbeater.cc:347] Connected to a master server at 127.5.77.254:44385
I20260924 12:48:45.012677 14139 heartbeater.cc:464] Registering TS with master...
I20260924 12:48:45.013509 14139 heartbeater.cc:511] Master 127.5.77.254:44385 requested a full tablet report, sending...
I20260924 12:48:45.015082 13822 ts_manager.cc:194] Registered new tserver with Master: 52fbcc075f81452e8ebea5dfa19c9aa7 (127.5.77.194:39559)
I20260924 12:48:45.015866 13822 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.194:48269
W20260924 12:48:45.326198 14142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:45.326496 14142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:45.326617 14142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:45.333878 14142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:45.334061 14142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.195
I20260924 12:48:45.343451 14142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:42361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.77.195
--webserver_port=0
--tserver_master_addrs=127.5.77.254:44385
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.195
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:45.344738 14142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:45.346426 14142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:45.359201 14147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:45.359157 14148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:45.359136 14150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:45.359982 14142 server_base.cc:1034] running on GCE node
I20260924 12:48:45.360581 14142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:45.361532 14142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:45.362756 14142 hybrid_clock.cc:648] HybridClock initialized: now 1790254125362680 us; error 49 us; skew 500 ppm
I20260924 12:48:45.363071 14142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:45.369673 14142 webserver.cc:460] Webserver started at http://127.5.77.195:36663/ using document root <none> and password file <none>
I20260924 12:48:45.370348 14142 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:45.370505 14142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:45.370812 14142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:45.373370 14142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "06404c97e0a74e03b6688dda9b5b31d4"
format_stamp: "Formatted at 2026-09-24 12:48:45 on dist-test-slave-b8q3"
I20260924 12:48:45.374085 14142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "06404c97e0a74e03b6688dda9b5b31d4"
format_stamp: "Formatted at 2026-09-24 12:48:45 on dist-test-slave-b8q3"
I20260924 12:48:45.379129 14142 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260924 12:48:45.382336 14156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:45.383685 14142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:48:45.383829 14142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "06404c97e0a74e03b6688dda9b5b31d4"
format_stamp: "Formatted at 2026-09-24 12:48:45 on dist-test-slave-b8q3"
I20260924 12:48:45.383988 14142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:45.404851 14142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:45.405669 14142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:45.405943 14142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:45.406916 14142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:45.408367 14142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:48:45.408439 14142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:45.408547 14142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:48:45.408604 14142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:45.437552 14142 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.195:35607
I20260924 12:48:45.437583 14268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.195:35607 every 8 connection(s)
I20260924 12:48:45.438800 14142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260924 12:48:45.439098 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 14142
I20260924 12:48:45.439253 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260924 12:48:45.454047 14269 heartbeater.cc:347] Connected to a master server at 127.5.77.254:44385
I20260924 12:48:45.454447 14269 heartbeater.cc:464] Registering TS with master...
I20260924 12:48:45.455556 14269 heartbeater.cc:511] Master 127.5.77.254:44385 requested a full tablet report, sending...
I20260924 12:48:45.457218 13822 ts_manager.cc:194] Registered new tserver with Master: 06404c97e0a74e03b6688dda9b5b31d4 (127.5.77.195:35607)
I20260924 12:48:45.458300 13822 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.195:45383
I20260924 12:48:45.467836 5431 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:48:45.484906 13822 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36036:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20260924 12:48:45.492868 13822 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:48:45.521438 14074 tablet_service.cc:1467] Processing CreateTablet for tablet 9e35d11aafd0452fae92273a7418fa26 (DEFAULT_TABLE table=TestTable [id=6e4ba2d08c8044659feb30817713acab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:48:45.521517 14204 tablet_service.cc:1467] Processing CreateTablet for tablet 9e35d11aafd0452fae92273a7418fa26 (DEFAULT_TABLE table=TestTable [id=6e4ba2d08c8044659feb30817713acab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:48:45.521975 13943 tablet_service.cc:1467] Processing CreateTablet for tablet 9e35d11aafd0452fae92273a7418fa26 (DEFAULT_TABLE table=TestTable [id=6e4ba2d08c8044659feb30817713acab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:48:45.522792 14074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e35d11aafd0452fae92273a7418fa26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:45.522980 14204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e35d11aafd0452fae92273a7418fa26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:45.523123 13943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e35d11aafd0452fae92273a7418fa26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:45.533018 14287 tablet_bootstrap.cc:492] T 9e35d11aafd0452fae92273a7418fa26 P 52fbcc075f81452e8ebea5dfa19c9aa7: Bootstrap starting.
I20260924 12:48:45.536492 14287 tablet_bootstrap.cc:654] T 9e35d11aafd0452fae92273a7418fa26 P 52fbcc075f81452e8ebea5dfa19c9aa7: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:45.537647 14287 log.cc:824] T 9e35d11aafd0452fae92273a7418fa26 P 52fbcc075f81452e8ebea5dfa19c9aa7: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:45.540153 14287 tablet_bootstrap.cc:492] T 9e35d11aafd0452fae92273a7418fa26 P 52fbcc075f81452e8ebea5dfa19c9aa7: No bootstrap required, opened a new log
I20260924 12:48:45.540381 14287 ts_tablet_manager.cc:1397] T 9e35d11aafd0452fae92273a7418fa26 P 52fbcc075f81452e8ebea5dfa19c9aa7: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260924 12:48:45.542804 14289 tablet_bootstrap.cc:492] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4: Bootstrap starting.
I20260924 12:48:45.543231 14288 tablet_bootstrap.cc:492] T 9e35d11aafd0452fae92273a7418fa26 P 088892440d344867b94ec0ad3e8346a8: Bootstrap starting.
I20260924 12:48:45.546869 14288 tablet_bootstrap.cc:654] T 9e35d11aafd0452fae92273a7418fa26 P 088892440d344867b94ec0ad3e8346a8: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:45.547519 14289 tablet_bootstrap.cc:654] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:45.548278 14288 log.cc:824] T 9e35d11aafd0452fae92273a7418fa26 P 088892440d344867b94ec0ad3e8346a8: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:45.548882 14289 log.cc:824] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:45.551034 14287 raft_consensus.cc:348] T 9e35d11aafd0452fae92273a7418fa26 P 52fbcc075f81452e8ebea5dfa19c9aa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39559 } } peers { permanent_uuid: "06404c97e0a74e03b6688dda9b5b31d4" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35607 } } peers { permanent_uuid: "088892440d344867b94ec0ad3e8346a8" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37165 } }
I20260924 12:48:45.551640 14287 raft_consensus.cc:374] T 9e35d11aafd0452fae92273a7418fa26 P 52fbcc075f81452e8ebea5dfa19c9aa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:48:45.551785 14287 raft_consensus.cc:729] T 9e35d11aafd0452fae92273a7418fa26 P 52fbcc075f81452e8ebea5dfa19c9aa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52fbcc075f81452e8ebea5dfa19c9aa7, State: Initialized, Role: FOLLOWER
I20260924 12:48:45.552631 14287 consensus_queue.cc:260] T 9e35d11aafd0452fae92273a7418fa26 P 52fbcc075f81452e8ebea5dfa19c9aa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39559 } } peers { permanent_uuid: "06404c97e0a74e03b6688dda9b5b31d4" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35607 } } peers { permanent_uuid: "088892440d344867b94ec0ad3e8346a8" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37165 } }
I20260924 12:48:45.557269 14139 heartbeater.cc:503] Master 127.5.77.254:44385 was elected leader, sending a full tablet report...
I20260924 12:48:45.557951 14287 ts_tablet_manager.cc:1428] T 9e35d11aafd0452fae92273a7418fa26 P 52fbcc075f81452e8ebea5dfa19c9aa7: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260924 12:48:45.558415 14288 tablet_bootstrap.cc:492] T 9e35d11aafd0452fae92273a7418fa26 P 088892440d344867b94ec0ad3e8346a8: No bootstrap required, opened a new log
I20260924 12:48:45.558712 14288 ts_tablet_manager.cc:1397] T 9e35d11aafd0452fae92273a7418fa26 P 088892440d344867b94ec0ad3e8346a8: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.006s
I20260924 12:48:45.561796 14289 tablet_bootstrap.cc:492] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4: No bootstrap required, opened a new log
I20260924 12:48:45.562063 14289 ts_tablet_manager.cc:1397] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.008s
I20260924 12:48:45.566773 14288 raft_consensus.cc:348] T 9e35d11aafd0452fae92273a7418fa26 P 088892440d344867b94ec0ad3e8346a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39559 } } peers { permanent_uuid: "06404c97e0a74e03b6688dda9b5b31d4" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35607 } } peers { permanent_uuid: "088892440d344867b94ec0ad3e8346a8" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37165 } }
I20260924 12:48:45.567123 14288 raft_consensus.cc:374] T 9e35d11aafd0452fae92273a7418fa26 P 088892440d344867b94ec0ad3e8346a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:48:45.567217 14288 raft_consensus.cc:729] T 9e35d11aafd0452fae92273a7418fa26 P 088892440d344867b94ec0ad3e8346a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088892440d344867b94ec0ad3e8346a8, State: Initialized, Role: FOLLOWER
I20260924 12:48:45.567782 14288 consensus_queue.cc:260] T 9e35d11aafd0452fae92273a7418fa26 P 088892440d344867b94ec0ad3e8346a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39559 } } peers { permanent_uuid: "06404c97e0a74e03b6688dda9b5b31d4" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35607 } } peers { permanent_uuid: "088892440d344867b94ec0ad3e8346a8" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37165 } }
I20260924 12:48:45.568776 14008 heartbeater.cc:503] Master 127.5.77.254:44385 was elected leader, sending a full tablet report...
I20260924 12:48:45.569448 14288 ts_tablet_manager.cc:1428] T 9e35d11aafd0452fae92273a7418fa26 P 088892440d344867b94ec0ad3e8346a8: Time spent starting tablet: real 0.011s user 0.004s sys 0.006s
I20260924 12:48:45.569367 14289 raft_consensus.cc:348] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39559 } } peers { permanent_uuid: "06404c97e0a74e03b6688dda9b5b31d4" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35607 } } peers { permanent_uuid: "088892440d344867b94ec0ad3e8346a8" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37165 } }
I20260924 12:48:45.569761 14289 raft_consensus.cc:374] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:48:45.569860 14289 raft_consensus.cc:729] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06404c97e0a74e03b6688dda9b5b31d4, State: Initialized, Role: FOLLOWER
I20260924 12:48:45.570417 14289 consensus_queue.cc:260] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39559 } } peers { permanent_uuid: "06404c97e0a74e03b6688dda9b5b31d4" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35607 } } peers { permanent_uuid: "088892440d344867b94ec0ad3e8346a8" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37165 } }
I20260924 12:48:45.571455 14269 heartbeater.cc:503] Master 127.5.77.254:44385 was elected leader, sending a full tablet report...
I20260924 12:48:45.571931 14289 ts_tablet_manager.cc:1428] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4: Time spent starting tablet: real 0.010s user 0.010s sys 0.001s
I20260924 12:48:45.587664 14295 raft_consensus.cc:482] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:48:45.587931 14295 raft_consensus.cc:504] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39559 } } peers { permanent_uuid: "06404c97e0a74e03b6688dda9b5b31d4" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35607 } } peers { permanent_uuid: "088892440d344867b94ec0ad3e8346a8" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37165 } }
I20260924 12:48:45.589596 14295 leader_election.cc:288] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52fbcc075f81452e8ebea5dfa19c9aa7 (127.5.77.194:39559), 088892440d344867b94ec0ad3e8346a8 (127.5.77.193:37165)
I20260924 12:48:45.602619 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "06404c97e0a74e03b6688dda9b5b31d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: true
I20260924 12:48:45.602978 14094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "06404c97e0a74e03b6688dda9b5b31d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" is_pre_election: true
I20260924 12:48:45.603250 13963 raft_consensus.cc:2454] T 9e35d11aafd0452fae92273a7418fa26 P 088892440d344867b94ec0ad3e8346a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06404c97e0a74e03b6688dda9b5b31d4 in term 0.
I20260924 12:48:45.603461 14094 raft_consensus.cc:2454] T 9e35d11aafd0452fae92273a7418fa26 P 52fbcc075f81452e8ebea5dfa19c9aa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06404c97e0a74e03b6688dda9b5b31d4 in term 0.
I20260924 12:48:45.604029 14157 leader_election.cc:302] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06404c97e0a74e03b6688dda9b5b31d4, 088892440d344867b94ec0ad3e8346a8; no voters:
I20260924 12:48:45.604498 14295 raft_consensus.cc:2781] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:48:45.604627 14295 raft_consensus.cc:482] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:48:45.604732 14295 raft_consensus.cc:3037] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:45.606757 14295 raft_consensus.cc:504] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39559 } } peers { permanent_uuid: "06404c97e0a74e03b6688dda9b5b31d4" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35607 } } peers { permanent_uuid: "088892440d344867b94ec0ad3e8346a8" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37165 } }
I20260924 12:48:45.607321 14295 leader_election.cc:288] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4 [CANDIDATE]: Term 1 election: Requested vote from peers 52fbcc075f81452e8ebea5dfa19c9aa7 (127.5.77.194:39559), 088892440d344867b94ec0ad3e8346a8 (127.5.77.193:37165)
I20260924 12:48:45.607829 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "06404c97e0a74e03b6688dda9b5b31d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8"
I20260924 12:48:45.607829 14094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "06404c97e0a74e03b6688dda9b5b31d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7"
I20260924 12:48:45.608038 14094 raft_consensus.cc:3037] T 9e35d11aafd0452fae92273a7418fa26 P 52fbcc075f81452e8ebea5dfa19c9aa7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:45.608038 13963 raft_consensus.cc:3037] T 9e35d11aafd0452fae92273a7418fa26 P 088892440d344867b94ec0ad3e8346a8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:45.610954 13963 raft_consensus.cc:2454] T 9e35d11aafd0452fae92273a7418fa26 P 088892440d344867b94ec0ad3e8346a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06404c97e0a74e03b6688dda9b5b31d4 in term 1.
I20260924 12:48:45.610954 14094 raft_consensus.cc:2454] T 9e35d11aafd0452fae92273a7418fa26 P 52fbcc075f81452e8ebea5dfa19c9aa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06404c97e0a74e03b6688dda9b5b31d4 in term 1.
I20260924 12:48:45.611347 14157 leader_election.cc:302] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06404c97e0a74e03b6688dda9b5b31d4, 088892440d344867b94ec0ad3e8346a8; no voters:
I20260924 12:48:45.611635 14295 raft_consensus.cc:2781] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:45.612061 14295 raft_consensus.cc:686] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4 [term 1 LEADER]: Becoming Leader. State: Replica: 06404c97e0a74e03b6688dda9b5b31d4, State: Running, Role: LEADER
I20260924 12:48:45.612583 14295 consensus_queue.cc:237] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39559 } } peers { permanent_uuid: "06404c97e0a74e03b6688dda9b5b31d4" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35607 } } peers { permanent_uuid: "088892440d344867b94ec0ad3e8346a8" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37165 } }
I20260924 12:48:45.616766 13821 catalog_manager.cc:5495] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06404c97e0a74e03b6688dda9b5b31d4 (127.5.77.195). New cstate: current_term: 1 leader_uuid: "06404c97e0a74e03b6688dda9b5b31d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06404c97e0a74e03b6688dda9b5b31d4" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 35607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088892440d344867b94ec0ad3e8346a8" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37165 } health_report { overall_health: UNKNOWN } } }
I20260924 12:48:45.633755 5431 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:48:45.635926 5431 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 088892440d344867b94ec0ad3e8346a8 to finish bootstrapping
I20260924 12:48:45.644392 5431 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 52fbcc075f81452e8ebea5dfa19c9aa7 to finish bootstrapping
I20260924 12:48:45.651635 5431 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 06404c97e0a74e03b6688dda9b5b31d4 to finish bootstrapping
I20260924 12:48:45.660179 14003 generic_service.cc:175] {username='slave'} at 127.0.0.1:41104 changed flags via RPC: log_inject_latency from 'false' to 'false'
I20260924 12:48:45.661660 14134 generic_service.cc:175] {username='slave'} at 127.0.0.1:36174 changed flags via RPC: log_inject_latency from 'false' to 'false'
I20260924 12:48:45.663012 14264 generic_service.cc:175] {username='slave'} at 127.0.0.1:53556 changed flags via RPC: log_inject_latency from 'false' to 'false'
I20260924 12:48:45.678857 13963 raft_consensus.cc:1261] T 9e35d11aafd0452fae92273a7418fa26 P 088892440d344867b94ec0ad3e8346a8 [term 1 FOLLOWER]: Refusing update from remote peer 06404c97e0a74e03b6688dda9b5b31d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:48:45.678921 14094 raft_consensus.cc:1261] T 9e35d11aafd0452fae92273a7418fa26 P 52fbcc075f81452e8ebea5dfa19c9aa7 [term 1 FOLLOWER]: Refusing update from remote peer 06404c97e0a74e03b6688dda9b5b31d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:48:45.679728 14297 consensus_queue.cc:1035] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 39559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:48:45.680070 14295 consensus_queue.cc:1035] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088892440d344867b94ec0ad3e8346a8" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:48:45.689754 14301 mvcc.cc:205] Tried to move back new op lower bound from 7332880898771910656 to 7332880898512748544. Current Snapshot: MvccSnapshot[applied={T|T < 7332880898771910656}]
I20260924 12:48:45.690686 14303 mvcc.cc:205] Tried to move back new op lower bound from 7332880898771910656 to 7332880898512748544. Current Snapshot: MvccSnapshot[applied={T|T < 7332880898771910656}]
I20260924 12:48:45.693989 14302 mvcc.cc:205] Tried to move back new op lower bound from 7332880898771910656 to 7332880898512748544. Current Snapshot: MvccSnapshot[applied={T|T < 7332880898771910656}]
I20260924 12:48:45.709870 14264 generic_service.cc:175] {username='slave'} at 127.0.0.1:53556 changed flags via RPC: log_inject_latency_ms_mean from '100' to '1200'
I20260924 12:48:45.711022 14264 generic_service.cc:175] {username='slave'} at 127.0.0.1:53556 changed flags via RPC: log_inject_latency_ms_stddev from '100' to '0'
I20260924 12:48:45.711997 14264 generic_service.cc:175] {username='slave'} at 127.0.0.1:53556 changed flags via RPC: log_inject_latency from 'false' to 'true'
I20260924 12:48:45.712944 14003 generic_service.cc:175] {username='slave'} at 127.0.0.1:41104 changed flags via RPC: log_inject_latency_ms_mean from '100' to '600'
I20260924 12:48:45.713752 14003 generic_service.cc:175] {username='slave'} at 127.0.0.1:41104 changed flags via RPC: log_inject_latency_ms_stddev from '100' to '0'
I20260924 12:48:45.714570 14003 generic_service.cc:175] {username='slave'} at 127.0.0.1:41104 changed flags via RPC: log_inject_latency from 'false' to 'true'
I20260924 12:48:45.714939 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 14012
I20260924 12:48:45.715755 14134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:45.803009 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 14012
I20260924 12:48:45.820220 5431 test_util.cc:268] Using random seed: 87697682
I20260924 12:48:45.834280 5431 test_workload.cc:401] TestWorkload: Skipping table creation because table TestTable already exists
W20260924 12:48:45.840544 14158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.77.194:39559: connect: Connection refused (error 111)
W20260924 12:48:45.842196 14158 consensus_peers.cc:487] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4 -> Peer 52fbcc075f81452e8ebea5dfa19c9aa7 (127.5.77.194:39559): Couldn't send request to peer 52fbcc075f81452e8ebea5dfa19c9aa7. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.194:39559: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 12:48:45.846873 14298 log.cc:508] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4: Injecting 1200ms of latency in SegmentAllocator::Sync()
W20260924 12:48:45.848575 14305 log.cc:508] T 9e35d11aafd0452fae92273a7418fa26 P 088892440d344867b94ec0ad3e8346a8: Injecting 600ms of latency in SegmentAllocator::Sync()
I20260924 12:48:45.886111 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 2 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:45.889199 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 2 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:45.941907 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 3 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:45.944253 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 3 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:45.996825 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 4 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:45.999512 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 4 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:46.052100 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 5 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:46.054409 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 5 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:46.106951 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 6 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:46.109256 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 6 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:46.161799 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 7 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:46.164271 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 7 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:46.216964 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 8 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:46.219269 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 8 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:46.271621 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 9 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:46.273886 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 9 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:46.326530 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 10 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:46.328929 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 10 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:46.381618 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 11 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:46.383942 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 11 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:46.436738 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 12 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:46.439671 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 12 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:46.492910 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 13 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:46.495302 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 13 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:46.547822 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 14 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:46.550392 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 14 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:46.603329 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 15 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:46.606446 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 15 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:46.659205 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 16 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:46.661546 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 16 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:46.714246 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 17 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:46.716784 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 17 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:46.769245 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 18 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:46.771720 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 18 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:46.824286 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 19 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:46.826848 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 19 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:46.879405 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 20 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:46.881676 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 20 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:46.934340 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 21 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:46.936945 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 21 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
W20260924 12:48:46.937588 14158 consensus_peers.cc:487] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4 -> Peer 52fbcc075f81452e8ebea5dfa19c9aa7 (127.5.77.194:39559): Couldn't send request to peer 52fbcc075f81452e8ebea5dfa19c9aa7. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.194:39559: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260924 12:48:46.989678 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 22 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:46.992146 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 22 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:47.044997 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 23 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:47.047473 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 23 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:47.049008 14319 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:53566 (ReqId={client: c0322e364c3e457eb789fcc50c4ba5f8, seq_no=0, attempt_no=0}) took 1204 ms. Trace:
I20260924 12:48:47.049193 14319 rpcz_store.cc:276] 0924 12:48:45.844556 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:48:45.844618 (+ 62us) service_pool.cc:227] Handling call
0924 12:48:47.048947 (+1204329us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:48:45.845747 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 9e35d11aafd0452fae92273a7418fa26
0924 12:48:45.845851 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:48:45.845856 (+ 5us) write_op.cc:435] Acquired schema lock
0924 12:48:45.845858 (+ 2us) tablet.cc:660] Decoding operations
0924 12:48:45.845932 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:48:45.845943 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0924 12:48:45.845946 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:48:45.845988 (+ 42us) tablet.cc:699] Row locks acquired
0924 12:48:45.845989 (+ 1us) write_op.cc:260] PREPARE: finished
0924 12:48:45.846108 (+ 119us) write_op.cc:270] Start()
0924 12:48:45.846137 (+ 29us) write_op.cc:276] Timestamp: P: 1790254125846045 usec, L: 0
0924 12:48:45.846140 (+ 3us) op_driver.cc:331] REPLICATION: starting
0924 12:48:45.846407 (+ 267us) log.cc:842] Serialized 345 byte log entry
0924 12:48:47.047370 (+1200963us) op_driver.cc:444] REPLICATION: finished
0924 12:48:47.047762 (+ 392us) write_op.cc:301] APPLY: starting
0924 12:48:47.047782 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0924 12:48:47.047827 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0924 12:48:47.047883 (+ 56us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:48:47.048008 (+ 125us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:48:47.048046 (+ 38us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:48:47.048054 (+ 8us) write_op.cc:312] APPLY: finished
0924 12:48:47.048200 (+ 146us) log.cc:842] Serialized 22 byte log entry
0924 12:48:47.048551 (+ 351us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:48:47.048578 (+ 27us) write_op.cc:454] Released schema lock
0924 12:48:47.048588 (+ 10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":325,"num_ops":1,"prepare.queue_time_us":50,"prepare.run_cpu_time_us":817,"prepare.run_wall_time_us":818,"raft.queue_time_us":107,"raft.run_cpu_time_us":257,"raft.run_wall_time_us":258,"replication_time_us":1201206,"thread_start_us":257,"threads_started":1}]]}
I20260924 12:48:47.100762 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 24 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:47.103339 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 24 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:47.155918 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 25 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:47.158380 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 25 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:47.211143 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 26 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:47.213637 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 26 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
W20260924 12:48:47.254544 14298 log.cc:508] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4: Injecting 1200ms of latency in SegmentAllocator::Sync()
W20260924 12:48:47.256307 14305 log.cc:508] T 9e35d11aafd0452fae92273a7418fa26 P 088892440d344867b94ec0ad3e8346a8: Injecting 600ms of latency in SegmentAllocator::Sync()
I20260924 12:48:47.266319 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 27 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:47.268694 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 27 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:47.321318 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 28 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:47.323688 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 28 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:47.376245 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 29 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:47.378697 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 29 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:47.431349 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 30 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:47.433703 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 30 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:47.486353 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 31 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:47.488801 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 31 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:47.541425 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 32 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:47.543998 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 32 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:47.596704 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 33 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:47.599157 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 33 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:47.651887 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 34 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:47.654644 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 34 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:47.707116 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 35 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:47.709445 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 35 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:47.762048 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 36 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:47.764544 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 36 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:47.817246 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 37 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:47.819692 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 37 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:47.872397 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 38 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:47.874933 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 38 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:47.927531 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 39 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:47.930020 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 39 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
W20260924 12:48:47.981652 14158 consensus_peers.cc:487] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4 -> Peer 52fbcc075f81452e8ebea5dfa19c9aa7 (127.5.77.194:39559): Couldn't send request to peer 52fbcc075f81452e8ebea5dfa19c9aa7. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.194:39559: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260924 12:48:47.982726 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 40 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:47.985245 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 40 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:48.037917 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 41 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:48.040501 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 41 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:48.093247 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 42 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:48.095633 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 42 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:48.148259 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 43 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:48.150841 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 43 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:48.203555 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 44 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:48.205994 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 44 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:48.258598 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 45 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:48.261166 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 45 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:48.314005 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 46 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:48.316507 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 46 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:48.369283 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 47 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:48.371768 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 47 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:48.424510 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 48 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:48.426889 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 48 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:48.456163 14325 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:53566 (ReqId={client: c0322e364c3e457eb789fcc50c4ba5f8, seq_no=1, attempt_no=0}) took 1203 ms. Trace:
I20260924 12:48:48.456372 14325 rpcz_store.cc:276] 0924 12:48:47.252225 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:48:47.252392 (+ 167us) service_pool.cc:227] Handling call
0924 12:48:48.456068 (+1203676us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:48:47.253530 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 9e35d11aafd0452fae92273a7418fa26
0924 12:48:47.253660 (+ 130us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:48:47.253667 (+ 7us) write_op.cc:435] Acquired schema lock
0924 12:48:47.253669 (+ 2us) tablet.cc:660] Decoding operations
0924 12:48:47.253756 (+ 87us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:48:47.253767 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0924 12:48:47.253771 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:48:47.253822 (+ 51us) tablet.cc:699] Row locks acquired
0924 12:48:47.253824 (+ 2us) write_op.cc:260] PREPARE: finished
0924 12:48:47.253897 (+ 73us) write_op.cc:270] Start()
0924 12:48:47.253939 (+ 42us) write_op.cc:276] Timestamp: P: 1790254127253855 usec, L: 0
0924 12:48:47.253942 (+ 3us) op_driver.cc:331] REPLICATION: starting
0924 12:48:47.254162 (+ 220us) log.cc:842] Serialized 354 byte log entry
0924 12:48:48.455061 (+1200899us) op_driver.cc:444] REPLICATION: finished
0924 12:48:48.455504 (+ 443us) write_op.cc:301] APPLY: starting
0924 12:48:48.455521 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0924 12:48:48.455561 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0924 12:48:48.455614 (+ 53us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:48:48.455699 (+ 85us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:48:48.455728 (+ 29us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:48:48.455733 (+ 5us) write_op.cc:312] APPLY: finished
0924 12:48:48.455811 (+ 78us) log.cc:842] Serialized 22 byte log entry
0924 12:48:48.455929 (+ 118us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:48:48.455952 (+ 23us) write_op.cc:454] Released schema lock
0924 12:48:48.455959 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":370,"num_ops":1,"prepare.queue_time_us":411,"prepare.run_cpu_time_us":765,"prepare.run_wall_time_us":765,"raft.queue_time_us":113,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":293,"replication_time_us":1201089,"thread_start_us":527,"threads_started":2}]]}
I20260924 12:48:48.479678 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 49 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:48.482105 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 49 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:48.534745 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 50 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:48.537317 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 50 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:48.589895 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 51 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:48.592492 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 51 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:48.645251 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 52 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:48.647895 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 52 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
W20260924 12:48:48.660267 14298 log.cc:508] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4: Injecting 1200ms of latency in SegmentAllocator::Sync()
W20260924 12:48:48.662058 14305 log.cc:508] T 9e35d11aafd0452fae92273a7418fa26 P 088892440d344867b94ec0ad3e8346a8: Injecting 600ms of latency in SegmentAllocator::Sync()
I20260924 12:48:48.700572 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 53 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:48.702977 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 53 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:48.755447 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 54 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:48.757721 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 54 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:48.810295 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 55 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:48.812665 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 55 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:48.865266 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 56 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:48.867650 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 56 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:48.920310 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 57 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
W20260924 12:48:48.922901 14158 consensus_peers.cc:487] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4 -> Peer 52fbcc075f81452e8ebea5dfa19c9aa7 (127.5.77.194:39559): Couldn't send request to peer 52fbcc075f81452e8ebea5dfa19c9aa7. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.194:39559: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260924 12:48:48.923097 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 57 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:48.975517 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 58 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:48.977804 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 58 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:49.030294 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 59 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:49.032816 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 59 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:49.085404 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 60 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:49.087987 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 60 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:49.140740 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 61 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:49.143417 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 61 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:49.196267 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 62 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:49.198836 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 62 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:49.251478 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 63 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:49.253935 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 63 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:49.306686 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 64 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:49.309128 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 64 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:49.361727 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 65 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:49.364202 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 65 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:49.416957 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 66 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:49.419647 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 66 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:49.472383 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 67 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:49.474798 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 67 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:49.527443 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 68 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:49.529793 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 68 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:49.582444 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 69 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:49.584908 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 69 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:49.637454 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 70 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:49.639842 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 70 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:49.692276 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 71 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:49.694784 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 71 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:49.747344 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 72 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:49.749648 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 72 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:49.802348 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 73 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:49.804785 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 73 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
W20260924 12:48:49.848189 14158 consensus_peers.cc:487] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4 -> Peer 52fbcc075f81452e8ebea5dfa19c9aa7 (127.5.77.194:39559): Couldn't send request to peer 52fbcc075f81452e8ebea5dfa19c9aa7. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.194:39559: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260924 12:48:49.857298 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 74 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:49.859872 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 74 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:49.862125 14331 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:53566 (ReqId={client: c0322e364c3e457eb789fcc50c4ba5f8, seq_no=2, attempt_no=0}) took 1203 ms. Trace:
I20260924 12:48:49.862255 14331 rpcz_store.cc:276] 0924 12:48:48.658395 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:48:48.658505 (+ 110us) service_pool.cc:227] Handling call
0924 12:48:49.862069 (+1203564us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:48:48.659373 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 9e35d11aafd0452fae92273a7418fa26
0924 12:48:48.659487 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:48:48.659492 (+ 5us) write_op.cc:435] Acquired schema lock
0924 12:48:48.659494 (+ 2us) tablet.cc:660] Decoding operations
0924 12:48:48.659569 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:48:48.659579 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0924 12:48:48.659583 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:48:48.659637 (+ 54us) tablet.cc:699] Row locks acquired
0924 12:48:48.659638 (+ 1us) write_op.cc:260] PREPARE: finished
0924 12:48:48.659676 (+ 38us) write_op.cc:270] Start()
0924 12:48:48.659715 (+ 39us) write_op.cc:276] Timestamp: P: 1790254128659630 usec, L: 0
0924 12:48:48.659718 (+ 3us) op_driver.cc:331] REPLICATION: starting
0924 12:48:48.659904 (+ 186us) log.cc:842] Serialized 354 byte log entry
0924 12:48:49.860771 (+1200867us) op_driver.cc:444] REPLICATION: finished
0924 12:48:49.861161 (+ 390us) write_op.cc:301] APPLY: starting
0924 12:48:49.861181 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0924 12:48:49.861230 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0924 12:48:49.861271 (+ 41us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:48:49.861373 (+ 102us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:48:49.861412 (+ 39us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:48:49.861417 (+ 5us) write_op.cc:312] APPLY: finished
0924 12:48:49.861522 (+ 105us) log.cc:842] Serialized 22 byte log entry
0924 12:48:49.861933 (+ 411us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:48:49.861956 (+ 23us) write_op.cc:454] Released schema lock
0924 12:48:49.861965 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":330,"num_ops":1,"prepare.queue_time_us":359,"prepare.run_cpu_time_us":648,"prepare.run_wall_time_us":648,"raft.queue_time_us":101,"raft.run_cpu_time_us":309,"raft.run_wall_time_us":311,"replication_time_us":1201022,"thread_start_us":467,"threads_started":2}]]}
I20260924 12:48:49.912662 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 75 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:49.915102 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 75 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:49.967706 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 76 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:49.970239 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 76 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:50.023038 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 77 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:50.025864 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 77 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
W20260924 12:48:50.066504 14298 log.cc:508] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4: Injecting 1200ms of latency in SegmentAllocator::Sync()
W20260924 12:48:50.068459 14305 log.cc:508] T 9e35d11aafd0452fae92273a7418fa26 P 088892440d344867b94ec0ad3e8346a8: Injecting 600ms of latency in SegmentAllocator::Sync()
I20260924 12:48:50.078737 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 78 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:50.081127 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 78 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:50.133633 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 79 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:50.136111 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 79 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:50.188588 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 80 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:50.190887 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 80 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:50.243420 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 81 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:50.245832 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 81 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:50.298391 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 82 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:50.300791 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 82 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:50.353375 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 83 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:50.355854 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 83 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:50.408573 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 84 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:50.411017 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 84 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:50.463507 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 85 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:50.466071 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 85 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:50.518757 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 86 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:50.521341 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 86 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:50.573980 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 87 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:50.576342 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 87 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:50.628961 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 88 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:50.631500 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 88 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:50.684276 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 89 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:50.686734 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 89 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:50.739387 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 90 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:50.741925 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 90 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
W20260924 12:48:50.763389 14158 consensus_peers.cc:487] T 9e35d11aafd0452fae92273a7418fa26 P 06404c97e0a74e03b6688dda9b5b31d4 -> Peer 52fbcc075f81452e8ebea5dfa19c9aa7 (127.5.77.194:39559): Couldn't send request to peer 52fbcc075f81452e8ebea5dfa19c9aa7. Status: Network error: Client connection negotiation failed: client connection to 127.5.77.194:39559: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260924 12:48:50.794597 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 91 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:50.797008 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 91 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:50.849898 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 92 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:50.852363 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 92 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:50.904827 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 93 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:50.907066 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 93 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:50.959714 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 94 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:50.962090 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 94 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
W20260924 12:48:50.972805 14158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.77.194:39559: connect: Connection refused (error 111) [suppressed 25 similar messages]
I20260924 12:48:51.014922 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 95 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:51.017390 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 95 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:51.070181 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 96 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:51.072867 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 96 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:51.125459 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 97 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:51.128196 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 97 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:51.180877 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 98 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:51.183364 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 98 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:51.236032 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 99 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:51.238410 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 99 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:51.268220 14337 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:53566 (ReqId={client: c0322e364c3e457eb789fcc50c4ba5f8, seq_no=3, attempt_no=0}) took 1203 ms. Trace:
I20260924 12:48:51.268452 14337 rpcz_store.cc:276] 0924 12:48:50.064580 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 12:48:50.064725 (+ 145us) service_pool.cc:227] Handling call
0924 12:48:51.268146 (+1203421us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 12:48:50.065576 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 9e35d11aafd0452fae92273a7418fa26
0924 12:48:50.065684 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0924 12:48:50.065689 (+ 5us) write_op.cc:435] Acquired schema lock
0924 12:48:50.065691 (+ 2us) tablet.cc:660] Decoding operations
0924 12:48:50.065792 (+ 101us) write_op.cc:620] Acquiring the partition lock for write op
0924 12:48:50.065804 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0924 12:48:50.065807 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0924 12:48:50.065852 (+ 45us) tablet.cc:699] Row locks acquired
0924 12:48:50.065853 (+ 1us) write_op.cc:260] PREPARE: finished
0924 12:48:50.065891 (+ 38us) write_op.cc:270] Start()
0924 12:48:50.065932 (+ 41us) write_op.cc:276] Timestamp: P: 1790254130065857 usec, L: 0
0924 12:48:50.065935 (+ 3us) op_driver.cc:331] REPLICATION: starting
0924 12:48:50.066130 (+ 195us) log.cc:842] Serialized 354 byte log entry
0924 12:48:51.267059 (+1200929us) op_driver.cc:444] REPLICATION: finished
0924 12:48:51.267491 (+ 432us) write_op.cc:301] APPLY: starting
0924 12:48:51.267508 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0924 12:48:51.267544 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0924 12:48:51.267584 (+ 40us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 12:48:51.267699 (+ 115us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 12:48:51.267726 (+ 27us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 12:48:51.267731 (+ 5us) write_op.cc:312] APPLY: finished
0924 12:48:51.267871 (+ 140us) log.cc:842] Serialized 22 byte log entry
0924 12:48:51.268002 (+ 131us) write_op.cc:489] Releasing partition, row and schema locks
0924 12:48:51.268026 (+ 24us) write_op.cc:454] Released schema lock
0924 12:48:51.268034 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":365,"num_ops":1,"prepare.queue_time_us":348,"prepare.run_cpu_time_us":664,"prepare.run_wall_time_us":665,"raft.queue_time_us":96,"raft.run_cpu_time_us":352,"raft.run_wall_time_us":354,"replication_time_us":1201093,"thread_start_us":478,"threads_started":2}]]}
I20260924 12:48:51.291311 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 100 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:51.293745 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 100 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:51.346452 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 101 index: 0 } } ignore_live_leader: false dest_uuid: "06404c97e0a74e03b6688dda9b5b31d4" is_pre_election: false
I20260924 12:48:51.348953 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35d11aafd0452fae92273a7418fa26" candidate_uuid: "52fbcc075f81452e8ebea5dfa19c9aa7" candidate_term: 2 candidate_status { last_received { term: 101 index: 0 } } ignore_live_leader: false dest_uuid: "088892440d344867b94ec0ad3e8346a8" is_pre_election: false
I20260924 12:48:51.472025 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 13881
I20260924 12:48:51.473201 14003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:51.560194 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 13881
I20260924 12:48:51.574651 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 14142
I20260924 12:48:51.575663 14264 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:51.667869 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 14142
I20260924 12:48:51.681396 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 13791
I20260924 12:48:51.682514 13852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:51.786206 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 13791
2026-09-24T12:48:51Z chronyd exiting
[ OK ] RaftConsensusElectionITest.DisruptiveServerAndSlowWAL (8267 ms)
[----------] 12 tests from RaftConsensusElectionITest (190171 ms total)
[----------] 2 tests from NumReplicas/RaftConsensusNumLeadersMetricTest
[ RUN ] NumReplicas/RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric/0
I20260924 12:48:51.821378 5431 test_util.cc:268] Using random seed: 93698823
I20260924 12:48:51.823206 5431 ts_itest-base.cc:112] Starting cluster with:
I20260924 12:48:51.823292 5431 ts_itest-base.cc:113] --------------
I20260924 12:48:51.823331 5431 ts_itest-base.cc:114] 1 tablet servers
I20260924 12:48:51.823376 5431 ts_itest-base.cc:115] 3 replicas per TS
I20260924 12:48:51.823410 5431 ts_itest-base.cc:116] --------------
2026-09-24T12:48:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:48:51Z Disabled control of system clock
I20260924 12:48:51.861752 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.77.254:43539
--webserver_interface=127.5.77.254
--webserver_port=0
--builtin_ntp_servers=127.5.77.212:41689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.77.254:43539
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20260924 12:48:52.179319 14352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:52.179630 14352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:52.179704 14352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:52.186985 14352 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:48:52.187100 14352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:52.187139 14352 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:48:52.187201 14352 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:48:52.196451 14352 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:41689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.5.77.254:43539
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.77.254:43539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.77.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:52.197708 14352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:52.199430 14352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:52.210776 14357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:52.210793 14358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:52.210719 14360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:52.211066 14352 server_base.cc:1034] running on GCE node
I20260924 12:48:52.212162 14352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:52.213565 14352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:52.214772 14352 hybrid_clock.cc:648] HybridClock initialized: now 1790254132214710 us; error 43 us; skew 500 ppm
I20260924 12:48:52.215080 14352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:52.221058 14352 webserver.cc:460] Webserver started at http://127.5.77.254:33337/ using document root <none> and password file <none>
I20260924 12:48:52.221796 14352 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:52.221913 14352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:52.222221 14352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:52.224951 14352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f7b81ae49b8d4e528d30ac5e157ee9f8"
format_stamp: "Formatted at 2026-09-24 12:48:52 on dist-test-slave-b8q3"
I20260924 12:48:52.225651 14352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f7b81ae49b8d4e528d30ac5e157ee9f8"
format_stamp: "Formatted at 2026-09-24 12:48:52 on dist-test-slave-b8q3"
I20260924 12:48:52.230729 14352 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260924 12:48:52.234022 14366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:52.235437 14352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:48:52.235594 14352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f7b81ae49b8d4e528d30ac5e157ee9f8"
format_stamp: "Formatted at 2026-09-24 12:48:52 on dist-test-slave-b8q3"
I20260924 12:48:52.235769 14352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:52.264160 14352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:52.264981 14352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:52.265225 14352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:52.281215 14352 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.254:43539
I20260924 12:48:52.281244 14417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.254:43539 every 8 connection(s)
I20260924 12:48:52.282503 14352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260924 12:48:52.285355 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 14352
I20260924 12:48:52.285531 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260924 12:48:52.286257 14418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:52.300293 14418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7b81ae49b8d4e528d30ac5e157ee9f8: Bootstrap starting.
I20260924 12:48:52.303790 14418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7b81ae49b8d4e528d30ac5e157ee9f8: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:52.304775 14418 log.cc:824] T 00000000000000000000000000000000 P f7b81ae49b8d4e528d30ac5e157ee9f8: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:52.307233 14418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7b81ae49b8d4e528d30ac5e157ee9f8: No bootstrap required, opened a new log
I20260924 12:48:52.314774 14418 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7b81ae49b8d4e528d30ac5e157ee9f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b81ae49b8d4e528d30ac5e157ee9f8" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 43539 } }
I20260924 12:48:52.315150 14418 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7b81ae49b8d4e528d30ac5e157ee9f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:48:52.315238 14418 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7b81ae49b8d4e528d30ac5e157ee9f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7b81ae49b8d4e528d30ac5e157ee9f8, State: Initialized, Role: FOLLOWER
I20260924 12:48:52.315789 14418 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7b81ae49b8d4e528d30ac5e157ee9f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b81ae49b8d4e528d30ac5e157ee9f8" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 43539 } }
I20260924 12:48:52.315987 14418 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7b81ae49b8d4e528d30ac5e157ee9f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:48:52.316114 14418 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7b81ae49b8d4e528d30ac5e157ee9f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:48:52.316262 14418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7b81ae49b8d4e528d30ac5e157ee9f8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:52.318259 14418 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7b81ae49b8d4e528d30ac5e157ee9f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b81ae49b8d4e528d30ac5e157ee9f8" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 43539 } }
I20260924 12:48:52.318806 14418 leader_election.cc:302] T 00000000000000000000000000000000 P f7b81ae49b8d4e528d30ac5e157ee9f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7b81ae49b8d4e528d30ac5e157ee9f8; no voters:
I20260924 12:48:52.319437 14418 leader_election.cc:288] T 00000000000000000000000000000000 P f7b81ae49b8d4e528d30ac5e157ee9f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:48:52.319533 14423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7b81ae49b8d4e528d30ac5e157ee9f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:52.320199 14423 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7b81ae49b8d4e528d30ac5e157ee9f8 [term 1 LEADER]: Becoming Leader. State: Replica: f7b81ae49b8d4e528d30ac5e157ee9f8, State: Running, Role: LEADER
I20260924 12:48:52.320698 14423 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7b81ae49b8d4e528d30ac5e157ee9f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b81ae49b8d4e528d30ac5e157ee9f8" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 43539 } }
I20260924 12:48:52.321395 14418 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7b81ae49b8d4e528d30ac5e157ee9f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:48:52.323412 14425 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7b81ae49b8d4e528d30ac5e157ee9f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7b81ae49b8d4e528d30ac5e157ee9f8. Latest consensus state: current_term: 1 leader_uuid: "f7b81ae49b8d4e528d30ac5e157ee9f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b81ae49b8d4e528d30ac5e157ee9f8" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 43539 } } }
I20260924 12:48:52.323391 14424 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7b81ae49b8d4e528d30ac5e157ee9f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7b81ae49b8d4e528d30ac5e157ee9f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b81ae49b8d4e528d30ac5e157ee9f8" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 43539 } } }
I20260924 12:48:52.323796 14424 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7b81ae49b8d4e528d30ac5e157ee9f8 [sys.catalog]: This master's current role is: LEADER
I20260924 12:48:52.323796 14425 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7b81ae49b8d4e528d30ac5e157ee9f8 [sys.catalog]: This master's current role is: LEADER
I20260924 12:48:52.328338 14429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:48:52.333118 14429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:48:52.341702 14429 catalog_manager.cc:1347] Generated new cluster ID: 8014bdf52b38411cb8debe0ce496abe3
I20260924 12:48:52.341809 14429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:48:52.361582 14429 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:48:52.363047 14429 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:48:52.373123 14429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7b81ae49b8d4e528d30ac5e157ee9f8: Generated new TSK 0
I20260924 12:48:52.373961 14429 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:48:52.386663 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.193:0
--local_ip_for_outbound_sockets=127.5.77.193
--webserver_interface=127.5.77.193
--webserver_port=0
--tserver_master_addrs=127.5.77.254:43539
--builtin_ntp_servers=127.5.77.212:41689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20260924 12:48:52.715507 14442 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 12:48:52.715808 14442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:52.715893 14442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:52.715982 14442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:52.723628 14442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:52.723799 14442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.193
I20260924 12:48:52.733170 14442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:41689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.77.193
--webserver_port=0
--tserver_master_addrs=127.5.77.254:43539
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.193
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:52.734491 14442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:52.736186 14442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:52.748610 14447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:52.748680 14448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:52.748638 14450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:52.749265 14442 server_base.cc:1034] running on GCE node
I20260924 12:48:52.749965 14442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:52.750944 14442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:52.752184 14442 hybrid_clock.cc:648] HybridClock initialized: now 1790254132752109 us; error 42 us; skew 500 ppm
I20260924 12:48:52.752513 14442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:52.754923 14442 webserver.cc:460] Webserver started at http://127.5.77.193:42681/ using document root <none> and password file <none>
I20260924 12:48:52.755622 14442 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:52.755712 14442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:52.756004 14442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:52.758663 14442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "3402a9aa258142149fc95120884519c0"
format_stamp: "Formatted at 2026-09-24 12:48:52 on dist-test-slave-b8q3"
I20260924 12:48:52.759407 14442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "3402a9aa258142149fc95120884519c0"
format_stamp: "Formatted at 2026-09-24 12:48:52 on dist-test-slave-b8q3"
I20260924 12:48:52.764525 14442 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 12:48:52.767921 14456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:52.769439 14442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 12:48:52.769657 14442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "3402a9aa258142149fc95120884519c0"
format_stamp: "Formatted at 2026-09-24 12:48:52 on dist-test-slave-b8q3"
I20260924 12:48:52.769869 14442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:52.788688 14442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:52.789548 14442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:52.789842 14442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:52.790876 14442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:52.792348 14442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:48:52.792444 14442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:52.792537 14442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:48:52.792604 14442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:52.824790 14442 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.193:42193
I20260924 12:48:52.824852 14568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.193:42193 every 8 connection(s)
I20260924 12:48:52.826185 14442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260924 12:48:52.835273 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 14442
I20260924 12:48:52.835472 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260924 12:48:52.840840 14569 heartbeater.cc:347] Connected to a master server at 127.5.77.254:43539
I20260924 12:48:52.841135 14569 heartbeater.cc:464] Registering TS with master...
I20260924 12:48:52.841856 14569 heartbeater.cc:511] Master 127.5.77.254:43539 requested a full tablet report, sending...
I20260924 12:48:52.843833 14383 ts_manager.cc:194] Registered new tserver with Master: 3402a9aa258142149fc95120884519c0 (127.5.77.193:42193)
I20260924 12:48:52.845330 14383 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.193:48193
I20260924 12:48:52.851645 5431 external_mini_cluster.cc:930] 1 TS(s) registered with all masters
I20260924 12:48:52.866405 5431 external_mini_cluster.cc:930] 1 TS(s) registered with all masters
I20260924 12:48:52.866549 5431 test_util.cc:268] Using random seed: 94744013
I20260924 12:48:52.886596 14383 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53516:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 12:48:52.914347 14498 tablet_service.cc:1467] Processing CreateTablet for tablet f391e251916144f9827ad1ec9a82d25e (DEFAULT_TABLE table=test-workload [id=de6cca36396c44d8ad26103a798b98c8]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260924 12:48:52.914780 14495 tablet_service.cc:1467] Processing CreateTablet for tablet 693d9014ea8b4162b5e9acae939e0d5d (DEFAULT_TABLE table=test-workload [id=de6cca36396c44d8ad26103a798b98c8]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260924 12:48:52.914014 14501 tablet_service.cc:1467] Processing CreateTablet for tablet 778dc068540e42cca8fbee526ee7b03f (DEFAULT_TABLE table=test-workload [id=de6cca36396c44d8ad26103a798b98c8]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260924 12:48:52.914238 14503 tablet_service.cc:1467] Processing CreateTablet for tablet cd86ffc0f7c141cabdfe32601456adbd (DEFAULT_TABLE table=test-workload [id=de6cca36396c44d8ad26103a798b98c8]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260924 12:48:52.915043 14504 tablet_service.cc:1467] Processing CreateTablet for tablet 7df1b7ec436041a19c28b65271ecfd4d (DEFAULT_TABLE table=test-workload [id=de6cca36396c44d8ad26103a798b98c8]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260924 12:48:52.914865 14499 tablet_service.cc:1467] Processing CreateTablet for tablet 8fa30b54787449159eb687f1d9a38500 (DEFAULT_TABLE table=test-workload [id=de6cca36396c44d8ad26103a798b98c8]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260924 12:48:52.914404 14497 tablet_service.cc:1467] Processing CreateTablet for tablet edd31a12eb714c3f9b473fc6e45be023 (DEFAULT_TABLE table=test-workload [id=de6cca36396c44d8ad26103a798b98c8]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260924 12:48:52.914052 14496 tablet_service.cc:1467] Processing CreateTablet for tablet 9b25405e75f94c069e182584aa5a987a (DEFAULT_TABLE table=test-workload [id=de6cca36396c44d8ad26103a798b98c8]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260924 12:48:52.915000 14502 tablet_service.cc:1467] Processing CreateTablet for tablet 467fe78d496a40ddaced927da77f01a6 (DEFAULT_TABLE table=test-workload [id=de6cca36396c44d8ad26103a798b98c8]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260924 12:48:52.916364 14495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 693d9014ea8b4162b5e9acae939e0d5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:52.914813 14500 tablet_service.cc:1467] Processing CreateTablet for tablet 83425546b6d2468892765374a54f21ef (DEFAULT_TABLE table=test-workload [id=de6cca36396c44d8ad26103a798b98c8]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260924 12:48:52.916826 14502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 467fe78d496a40ddaced927da77f01a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:52.917028 14501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 778dc068540e42cca8fbee526ee7b03f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:52.917258 14496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b25405e75f94c069e182584aa5a987a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:52.917668 14499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fa30b54787449159eb687f1d9a38500. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:52.918161 14500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83425546b6d2468892765374a54f21ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:52.918668 14504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7df1b7ec436041a19c28b65271ecfd4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:52.919909 14497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edd31a12eb714c3f9b473fc6e45be023. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:52.920665 14498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f391e251916144f9827ad1ec9a82d25e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:52.921409 14503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd86ffc0f7c141cabdfe32601456adbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:52.942339 14588 tablet_bootstrap.cc:492] T 467fe78d496a40ddaced927da77f01a6 P 3402a9aa258142149fc95120884519c0: Bootstrap starting.
I20260924 12:48:52.945293 14588 tablet_bootstrap.cc:654] T 467fe78d496a40ddaced927da77f01a6 P 3402a9aa258142149fc95120884519c0: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:52.946273 14588 log.cc:824] T 467fe78d496a40ddaced927da77f01a6 P 3402a9aa258142149fc95120884519c0: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:52.948745 14588 tablet_bootstrap.cc:492] T 467fe78d496a40ddaced927da77f01a6 P 3402a9aa258142149fc95120884519c0: No bootstrap required, opened a new log
I20260924 12:48:52.949008 14588 ts_tablet_manager.cc:1397] T 467fe78d496a40ddaced927da77f01a6 P 3402a9aa258142149fc95120884519c0: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 12:48:52.956008 14588 raft_consensus.cc:348] T 467fe78d496a40ddaced927da77f01a6 P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:52.956406 14588 raft_consensus.cc:729] T 467fe78d496a40ddaced927da77f01a6 P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3402a9aa258142149fc95120884519c0, State: Initialized, Role: FOLLOWER
I20260924 12:48:52.956833 14588 consensus_queue.cc:260] T 467fe78d496a40ddaced927da77f01a6 P 3402a9aa258142149fc95120884519c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:52.957844 14569 heartbeater.cc:503] Master 127.5.77.254:43539 was elected leader, sending a full tablet report...
I20260924 12:48:52.958568 14588 ts_tablet_manager.cc:1428] T 467fe78d496a40ddaced927da77f01a6 P 3402a9aa258142149fc95120884519c0: Time spent starting tablet: real 0.009s user 0.006s sys 0.005s
I20260924 12:48:52.958937 14588 tablet_bootstrap.cc:492] T 778dc068540e42cca8fbee526ee7b03f P 3402a9aa258142149fc95120884519c0: Bootstrap starting.
I20260924 12:48:52.960505 14588 tablet_bootstrap.cc:654] T 778dc068540e42cca8fbee526ee7b03f P 3402a9aa258142149fc95120884519c0: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:52.962598 14588 tablet_bootstrap.cc:492] T 778dc068540e42cca8fbee526ee7b03f P 3402a9aa258142149fc95120884519c0: No bootstrap required, opened a new log
I20260924 12:48:52.962817 14588 ts_tablet_manager.cc:1397] T 778dc068540e42cca8fbee526ee7b03f P 3402a9aa258142149fc95120884519c0: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260924 12:48:52.963614 14588 raft_consensus.cc:348] T 778dc068540e42cca8fbee526ee7b03f P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:52.963790 14588 raft_consensus.cc:729] T 778dc068540e42cca8fbee526ee7b03f P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3402a9aa258142149fc95120884519c0, State: Initialized, Role: FOLLOWER
I20260924 12:48:52.963953 14588 consensus_queue.cc:260] T 778dc068540e42cca8fbee526ee7b03f P 3402a9aa258142149fc95120884519c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:52.964524 14588 ts_tablet_manager.cc:1428] T 778dc068540e42cca8fbee526ee7b03f P 3402a9aa258142149fc95120884519c0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 12:48:52.964778 14588 tablet_bootstrap.cc:492] T cd86ffc0f7c141cabdfe32601456adbd P 3402a9aa258142149fc95120884519c0: Bootstrap starting.
I20260924 12:48:52.965989 14588 tablet_bootstrap.cc:654] T cd86ffc0f7c141cabdfe32601456adbd P 3402a9aa258142149fc95120884519c0: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:52.968035 14588 tablet_bootstrap.cc:492] T cd86ffc0f7c141cabdfe32601456adbd P 3402a9aa258142149fc95120884519c0: No bootstrap required, opened a new log
I20260924 12:48:52.968196 14588 ts_tablet_manager.cc:1397] T cd86ffc0f7c141cabdfe32601456adbd P 3402a9aa258142149fc95120884519c0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 12:48:52.968883 14588 raft_consensus.cc:348] T cd86ffc0f7c141cabdfe32601456adbd P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:52.969096 14588 raft_consensus.cc:729] T cd86ffc0f7c141cabdfe32601456adbd P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3402a9aa258142149fc95120884519c0, State: Initialized, Role: FOLLOWER
I20260924 12:48:52.969287 14588 consensus_queue.cc:260] T cd86ffc0f7c141cabdfe32601456adbd P 3402a9aa258142149fc95120884519c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:52.969710 14588 ts_tablet_manager.cc:1428] T cd86ffc0f7c141cabdfe32601456adbd P 3402a9aa258142149fc95120884519c0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 12:48:52.969935 14588 tablet_bootstrap.cc:492] T 9b25405e75f94c069e182584aa5a987a P 3402a9aa258142149fc95120884519c0: Bootstrap starting.
I20260924 12:48:52.971323 14588 tablet_bootstrap.cc:654] T 9b25405e75f94c069e182584aa5a987a P 3402a9aa258142149fc95120884519c0: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:52.973362 14588 tablet_bootstrap.cc:492] T 9b25405e75f94c069e182584aa5a987a P 3402a9aa258142149fc95120884519c0: No bootstrap required, opened a new log
I20260924 12:48:52.973521 14588 ts_tablet_manager.cc:1397] T 9b25405e75f94c069e182584aa5a987a P 3402a9aa258142149fc95120884519c0: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260924 12:48:52.974169 14588 raft_consensus.cc:348] T 9b25405e75f94c069e182584aa5a987a P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:52.974365 14588 raft_consensus.cc:729] T 9b25405e75f94c069e182584aa5a987a P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3402a9aa258142149fc95120884519c0, State: Initialized, Role: FOLLOWER
I20260924 12:48:52.974555 14588 consensus_queue.cc:260] T 9b25405e75f94c069e182584aa5a987a P 3402a9aa258142149fc95120884519c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:52.974978 14588 ts_tablet_manager.cc:1428] T 9b25405e75f94c069e182584aa5a987a P 3402a9aa258142149fc95120884519c0: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 12:48:52.975252 14588 tablet_bootstrap.cc:492] T edd31a12eb714c3f9b473fc6e45be023 P 3402a9aa258142149fc95120884519c0: Bootstrap starting.
I20260924 12:48:52.976763 14588 tablet_bootstrap.cc:654] T edd31a12eb714c3f9b473fc6e45be023 P 3402a9aa258142149fc95120884519c0: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:52.978591 14588 tablet_bootstrap.cc:492] T edd31a12eb714c3f9b473fc6e45be023 P 3402a9aa258142149fc95120884519c0: No bootstrap required, opened a new log
I20260924 12:48:52.978727 14588 ts_tablet_manager.cc:1397] T edd31a12eb714c3f9b473fc6e45be023 P 3402a9aa258142149fc95120884519c0: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260924 12:48:52.979318 14588 raft_consensus.cc:348] T edd31a12eb714c3f9b473fc6e45be023 P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:52.979511 14588 raft_consensus.cc:729] T edd31a12eb714c3f9b473fc6e45be023 P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3402a9aa258142149fc95120884519c0, State: Initialized, Role: FOLLOWER
I20260924 12:48:52.979683 14588 consensus_queue.cc:260] T edd31a12eb714c3f9b473fc6e45be023 P 3402a9aa258142149fc95120884519c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:52.980175 14588 ts_tablet_manager.cc:1428] T edd31a12eb714c3f9b473fc6e45be023 P 3402a9aa258142149fc95120884519c0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 12:48:52.980415 14588 tablet_bootstrap.cc:492] T f391e251916144f9827ad1ec9a82d25e P 3402a9aa258142149fc95120884519c0: Bootstrap starting.
I20260924 12:48:52.981863 14588 tablet_bootstrap.cc:654] T f391e251916144f9827ad1ec9a82d25e P 3402a9aa258142149fc95120884519c0: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:52.983642 14588 tablet_bootstrap.cc:492] T f391e251916144f9827ad1ec9a82d25e P 3402a9aa258142149fc95120884519c0: No bootstrap required, opened a new log
I20260924 12:48:52.983778 14588 ts_tablet_manager.cc:1397] T f391e251916144f9827ad1ec9a82d25e P 3402a9aa258142149fc95120884519c0: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 12:48:52.984359 14588 raft_consensus.cc:348] T f391e251916144f9827ad1ec9a82d25e P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:52.984522 14588 raft_consensus.cc:729] T f391e251916144f9827ad1ec9a82d25e P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3402a9aa258142149fc95120884519c0, State: Initialized, Role: FOLLOWER
I20260924 12:48:52.984716 14588 consensus_queue.cc:260] T f391e251916144f9827ad1ec9a82d25e P 3402a9aa258142149fc95120884519c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:52.985163 14588 ts_tablet_manager.cc:1428] T f391e251916144f9827ad1ec9a82d25e P 3402a9aa258142149fc95120884519c0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 12:48:52.985381 14588 tablet_bootstrap.cc:492] T 7df1b7ec436041a19c28b65271ecfd4d P 3402a9aa258142149fc95120884519c0: Bootstrap starting.
I20260924 12:48:52.986589 14588 tablet_bootstrap.cc:654] T 7df1b7ec436041a19c28b65271ecfd4d P 3402a9aa258142149fc95120884519c0: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:52.988217 14588 tablet_bootstrap.cc:492] T 7df1b7ec436041a19c28b65271ecfd4d P 3402a9aa258142149fc95120884519c0: No bootstrap required, opened a new log
I20260924 12:48:52.988358 14588 ts_tablet_manager.cc:1397] T 7df1b7ec436041a19c28b65271ecfd4d P 3402a9aa258142149fc95120884519c0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 12:48:52.989013 14588 raft_consensus.cc:348] T 7df1b7ec436041a19c28b65271ecfd4d P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:52.989167 14588 raft_consensus.cc:729] T 7df1b7ec436041a19c28b65271ecfd4d P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3402a9aa258142149fc95120884519c0, State: Initialized, Role: FOLLOWER
I20260924 12:48:52.989329 14588 consensus_queue.cc:260] T 7df1b7ec436041a19c28b65271ecfd4d P 3402a9aa258142149fc95120884519c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:52.989720 14588 ts_tablet_manager.cc:1428] T 7df1b7ec436041a19c28b65271ecfd4d P 3402a9aa258142149fc95120884519c0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 12:48:52.989946 14588 tablet_bootstrap.cc:492] T 83425546b6d2468892765374a54f21ef P 3402a9aa258142149fc95120884519c0: Bootstrap starting.
I20260924 12:48:52.991117 14588 tablet_bootstrap.cc:654] T 83425546b6d2468892765374a54f21ef P 3402a9aa258142149fc95120884519c0: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:52.992897 14588 tablet_bootstrap.cc:492] T 83425546b6d2468892765374a54f21ef P 3402a9aa258142149fc95120884519c0: No bootstrap required, opened a new log
I20260924 12:48:52.993043 14588 ts_tablet_manager.cc:1397] T 83425546b6d2468892765374a54f21ef P 3402a9aa258142149fc95120884519c0: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 12:48:52.993625 14588 raft_consensus.cc:348] T 83425546b6d2468892765374a54f21ef P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:52.993830 14588 raft_consensus.cc:729] T 83425546b6d2468892765374a54f21ef P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3402a9aa258142149fc95120884519c0, State: Initialized, Role: FOLLOWER
I20260924 12:48:52.993985 14588 consensus_queue.cc:260] T 83425546b6d2468892765374a54f21ef P 3402a9aa258142149fc95120884519c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:52.994385 14588 ts_tablet_manager.cc:1428] T 83425546b6d2468892765374a54f21ef P 3402a9aa258142149fc95120884519c0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 12:48:52.994629 14588 tablet_bootstrap.cc:492] T 693d9014ea8b4162b5e9acae939e0d5d P 3402a9aa258142149fc95120884519c0: Bootstrap starting.
I20260924 12:48:52.995970 14588 tablet_bootstrap.cc:654] T 693d9014ea8b4162b5e9acae939e0d5d P 3402a9aa258142149fc95120884519c0: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:52.997931 14588 tablet_bootstrap.cc:492] T 693d9014ea8b4162b5e9acae939e0d5d P 3402a9aa258142149fc95120884519c0: No bootstrap required, opened a new log
I20260924 12:48:52.998070 14588 ts_tablet_manager.cc:1397] T 693d9014ea8b4162b5e9acae939e0d5d P 3402a9aa258142149fc95120884519c0: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260924 12:48:52.998708 14588 raft_consensus.cc:348] T 693d9014ea8b4162b5e9acae939e0d5d P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:52.998898 14588 raft_consensus.cc:729] T 693d9014ea8b4162b5e9acae939e0d5d P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3402a9aa258142149fc95120884519c0, State: Initialized, Role: FOLLOWER
I20260924 12:48:52.999049 14588 consensus_queue.cc:260] T 693d9014ea8b4162b5e9acae939e0d5d P 3402a9aa258142149fc95120884519c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:52.999459 14588 ts_tablet_manager.cc:1428] T 693d9014ea8b4162b5e9acae939e0d5d P 3402a9aa258142149fc95120884519c0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 12:48:52.999702 14588 tablet_bootstrap.cc:492] T 8fa30b54787449159eb687f1d9a38500 P 3402a9aa258142149fc95120884519c0: Bootstrap starting.
I20260924 12:48:53.000919 14588 tablet_bootstrap.cc:654] T 8fa30b54787449159eb687f1d9a38500 P 3402a9aa258142149fc95120884519c0: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:53.002647 14588 tablet_bootstrap.cc:492] T 8fa30b54787449159eb687f1d9a38500 P 3402a9aa258142149fc95120884519c0: No bootstrap required, opened a new log
I20260924 12:48:53.002786 14588 ts_tablet_manager.cc:1397] T 8fa30b54787449159eb687f1d9a38500 P 3402a9aa258142149fc95120884519c0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 12:48:53.003357 14588 raft_consensus.cc:348] T 8fa30b54787449159eb687f1d9a38500 P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:53.003553 14588 raft_consensus.cc:729] T 8fa30b54787449159eb687f1d9a38500 P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3402a9aa258142149fc95120884519c0, State: Initialized, Role: FOLLOWER
I20260924 12:48:53.003736 14588 consensus_queue.cc:260] T 8fa30b54787449159eb687f1d9a38500 P 3402a9aa258142149fc95120884519c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:53.004154 14588 ts_tablet_manager.cc:1428] T 8fa30b54787449159eb687f1d9a38500 P 3402a9aa258142149fc95120884519c0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 12:48:53.049285 14524 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "693d9014ea8b4162b5e9acae939e0d5d"
dest_uuid: "3402a9aa258142149fc95120884519c0"
from {username='slave'} at 127.0.0.1:60182
I20260924 12:48:53.049600 14524 raft_consensus.cc:482] T 693d9014ea8b4162b5e9acae939e0d5d P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 12:48:53.049750 14524 raft_consensus.cc:3037] T 693d9014ea8b4162b5e9acae939e0d5d P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:53.051810 14524 raft_consensus.cc:504] T 693d9014ea8b4162b5e9acae939e0d5d P 3402a9aa258142149fc95120884519c0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:53.052320 14524 leader_election.cc:302] T 693d9014ea8b4162b5e9acae939e0d5d P 3402a9aa258142149fc95120884519c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3402a9aa258142149fc95120884519c0; no voters:
I20260924 12:48:53.052613 14524 leader_election.cc:288] T 693d9014ea8b4162b5e9acae939e0d5d P 3402a9aa258142149fc95120884519c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:48:53.052721 14590 raft_consensus.cc:2781] T 693d9014ea8b4162b5e9acae939e0d5d P 3402a9aa258142149fc95120884519c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:53.053315 14590 raft_consensus.cc:686] T 693d9014ea8b4162b5e9acae939e0d5d P 3402a9aa258142149fc95120884519c0 [term 1 LEADER]: Becoming Leader. State: Replica: 3402a9aa258142149fc95120884519c0, State: Running, Role: LEADER
I20260924 12:48:53.053848 14590 consensus_queue.cc:237] T 693d9014ea8b4162b5e9acae939e0d5d P 3402a9aa258142149fc95120884519c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:53.058398 14383 catalog_manager.cc:5495] T 693d9014ea8b4162b5e9acae939e0d5d P 3402a9aa258142149fc95120884519c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3402a9aa258142149fc95120884519c0 (127.5.77.193). New cstate: current_term: 1 leader_uuid: "3402a9aa258142149fc95120884519c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } health_report { overall_health: HEALTHY } } }
I20260924 12:48:53.066707 14524 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "83425546b6d2468892765374a54f21ef"
dest_uuid: "3402a9aa258142149fc95120884519c0"
from {username='slave'} at 127.0.0.1:60182
I20260924 12:48:53.066905 14524 raft_consensus.cc:482] T 83425546b6d2468892765374a54f21ef P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 12:48:53.066998 14524 raft_consensus.cc:3037] T 83425546b6d2468892765374a54f21ef P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:53.068801 14524 raft_consensus.cc:504] T 83425546b6d2468892765374a54f21ef P 3402a9aa258142149fc95120884519c0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:53.069031 14524 leader_election.cc:302] T 83425546b6d2468892765374a54f21ef P 3402a9aa258142149fc95120884519c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3402a9aa258142149fc95120884519c0; no voters:
I20260924 12:48:53.069169 14524 leader_election.cc:288] T 83425546b6d2468892765374a54f21ef P 3402a9aa258142149fc95120884519c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:48:53.069274 14590 raft_consensus.cc:2781] T 83425546b6d2468892765374a54f21ef P 3402a9aa258142149fc95120884519c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:53.069445 14590 raft_consensus.cc:686] T 83425546b6d2468892765374a54f21ef P 3402a9aa258142149fc95120884519c0 [term 1 LEADER]: Becoming Leader. State: Replica: 3402a9aa258142149fc95120884519c0, State: Running, Role: LEADER
I20260924 12:48:53.069676 14590 consensus_queue.cc:237] T 83425546b6d2468892765374a54f21ef P 3402a9aa258142149fc95120884519c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:53.071902 14383 catalog_manager.cc:5495] T 83425546b6d2468892765374a54f21ef P 3402a9aa258142149fc95120884519c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3402a9aa258142149fc95120884519c0 (127.5.77.193). New cstate: current_term: 1 leader_uuid: "3402a9aa258142149fc95120884519c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } health_report { overall_health: HEALTHY } } }
W20260924 12:48:53.077847 14570 tablet.cc:2369] T 467fe78d496a40ddaced927da77f01a6 P 3402a9aa258142149fc95120884519c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:48:53.080909 14524 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7df1b7ec436041a19c28b65271ecfd4d"
dest_uuid: "3402a9aa258142149fc95120884519c0"
from {username='slave'} at 127.0.0.1:60182
I20260924 12:48:53.081110 14524 raft_consensus.cc:482] T 7df1b7ec436041a19c28b65271ecfd4d P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 12:48:53.081262 14524 raft_consensus.cc:3037] T 7df1b7ec436041a19c28b65271ecfd4d P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:53.083165 14524 raft_consensus.cc:504] T 7df1b7ec436041a19c28b65271ecfd4d P 3402a9aa258142149fc95120884519c0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:53.083413 14524 leader_election.cc:302] T 7df1b7ec436041a19c28b65271ecfd4d P 3402a9aa258142149fc95120884519c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3402a9aa258142149fc95120884519c0; no voters:
I20260924 12:48:53.083593 14524 leader_election.cc:288] T 7df1b7ec436041a19c28b65271ecfd4d P 3402a9aa258142149fc95120884519c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:48:53.083683 14590 raft_consensus.cc:2781] T 7df1b7ec436041a19c28b65271ecfd4d P 3402a9aa258142149fc95120884519c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:53.083855 14590 raft_consensus.cc:686] T 7df1b7ec436041a19c28b65271ecfd4d P 3402a9aa258142149fc95120884519c0 [term 1 LEADER]: Becoming Leader. State: Replica: 3402a9aa258142149fc95120884519c0, State: Running, Role: LEADER
I20260924 12:48:53.084095 14590 consensus_queue.cc:237] T 7df1b7ec436041a19c28b65271ecfd4d P 3402a9aa258142149fc95120884519c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:53.086495 14383 catalog_manager.cc:5495] T 7df1b7ec436041a19c28b65271ecfd4d P 3402a9aa258142149fc95120884519c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3402a9aa258142149fc95120884519c0 (127.5.77.193). New cstate: current_term: 1 leader_uuid: "3402a9aa258142149fc95120884519c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } health_report { overall_health: HEALTHY } } }
I20260924 12:48:53.095079 14524 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8fa30b54787449159eb687f1d9a38500"
dest_uuid: "3402a9aa258142149fc95120884519c0"
from {username='slave'} at 127.0.0.1:60182
I20260924 12:48:53.095315 14524 raft_consensus.cc:482] T 8fa30b54787449159eb687f1d9a38500 P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 12:48:53.095425 14524 raft_consensus.cc:3037] T 8fa30b54787449159eb687f1d9a38500 P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:53.097224 14524 raft_consensus.cc:504] T 8fa30b54787449159eb687f1d9a38500 P 3402a9aa258142149fc95120884519c0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:53.097461 14524 leader_election.cc:302] T 8fa30b54787449159eb687f1d9a38500 P 3402a9aa258142149fc95120884519c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3402a9aa258142149fc95120884519c0; no voters:
I20260924 12:48:53.097630 14524 leader_election.cc:288] T 8fa30b54787449159eb687f1d9a38500 P 3402a9aa258142149fc95120884519c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:48:53.097719 14590 raft_consensus.cc:2781] T 8fa30b54787449159eb687f1d9a38500 P 3402a9aa258142149fc95120884519c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:53.097898 14590 raft_consensus.cc:686] T 8fa30b54787449159eb687f1d9a38500 P 3402a9aa258142149fc95120884519c0 [term 1 LEADER]: Becoming Leader. State: Replica: 3402a9aa258142149fc95120884519c0, State: Running, Role: LEADER
I20260924 12:48:53.098115 14590 consensus_queue.cc:237] T 8fa30b54787449159eb687f1d9a38500 P 3402a9aa258142149fc95120884519c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:53.100257 14383 catalog_manager.cc:5495] T 8fa30b54787449159eb687f1d9a38500 P 3402a9aa258142149fc95120884519c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3402a9aa258142149fc95120884519c0 (127.5.77.193). New cstate: current_term: 1 leader_uuid: "3402a9aa258142149fc95120884519c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } health_report { overall_health: HEALTHY } } }
I20260924 12:48:53.108904 14524 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "778dc068540e42cca8fbee526ee7b03f"
dest_uuid: "3402a9aa258142149fc95120884519c0"
from {username='slave'} at 127.0.0.1:60182
I20260924 12:48:53.109125 14524 raft_consensus.cc:482] T 778dc068540e42cca8fbee526ee7b03f P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 12:48:53.109225 14524 raft_consensus.cc:3037] T 778dc068540e42cca8fbee526ee7b03f P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:53.111112 14524 raft_consensus.cc:504] T 778dc068540e42cca8fbee526ee7b03f P 3402a9aa258142149fc95120884519c0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:53.111344 14524 leader_election.cc:302] T 778dc068540e42cca8fbee526ee7b03f P 3402a9aa258142149fc95120884519c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3402a9aa258142149fc95120884519c0; no voters:
I20260924 12:48:53.111481 14524 leader_election.cc:288] T 778dc068540e42cca8fbee526ee7b03f P 3402a9aa258142149fc95120884519c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:48:53.111608 14590 raft_consensus.cc:2781] T 778dc068540e42cca8fbee526ee7b03f P 3402a9aa258142149fc95120884519c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:53.111799 14590 raft_consensus.cc:686] T 778dc068540e42cca8fbee526ee7b03f P 3402a9aa258142149fc95120884519c0 [term 1 LEADER]: Becoming Leader. State: Replica: 3402a9aa258142149fc95120884519c0, State: Running, Role: LEADER
I20260924 12:48:53.112033 14590 consensus_queue.cc:237] T 778dc068540e42cca8fbee526ee7b03f P 3402a9aa258142149fc95120884519c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:53.114279 14383 catalog_manager.cc:5495] T 778dc068540e42cca8fbee526ee7b03f P 3402a9aa258142149fc95120884519c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3402a9aa258142149fc95120884519c0 (127.5.77.193). New cstate: current_term: 1 leader_uuid: "3402a9aa258142149fc95120884519c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } health_report { overall_health: HEALTHY } } }
I20260924 12:48:53.123086 14524 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "edd31a12eb714c3f9b473fc6e45be023"
dest_uuid: "3402a9aa258142149fc95120884519c0"
from {username='slave'} at 127.0.0.1:60182
I20260924 12:48:53.123311 14524 raft_consensus.cc:482] T edd31a12eb714c3f9b473fc6e45be023 P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 12:48:53.123456 14524 raft_consensus.cc:3037] T edd31a12eb714c3f9b473fc6e45be023 P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:53.125356 14524 raft_consensus.cc:504] T edd31a12eb714c3f9b473fc6e45be023 P 3402a9aa258142149fc95120884519c0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:53.125591 14524 leader_election.cc:302] T edd31a12eb714c3f9b473fc6e45be023 P 3402a9aa258142149fc95120884519c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3402a9aa258142149fc95120884519c0; no voters:
I20260924 12:48:53.125766 14524 leader_election.cc:288] T edd31a12eb714c3f9b473fc6e45be023 P 3402a9aa258142149fc95120884519c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:48:53.125880 14590 raft_consensus.cc:2781] T edd31a12eb714c3f9b473fc6e45be023 P 3402a9aa258142149fc95120884519c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:53.126096 14590 raft_consensus.cc:686] T edd31a12eb714c3f9b473fc6e45be023 P 3402a9aa258142149fc95120884519c0 [term 1 LEADER]: Becoming Leader. State: Replica: 3402a9aa258142149fc95120884519c0, State: Running, Role: LEADER
I20260924 12:48:53.126336 14590 consensus_queue.cc:237] T edd31a12eb714c3f9b473fc6e45be023 P 3402a9aa258142149fc95120884519c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:53.128896 14383 catalog_manager.cc:5495] T edd31a12eb714c3f9b473fc6e45be023 P 3402a9aa258142149fc95120884519c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3402a9aa258142149fc95120884519c0 (127.5.77.193). New cstate: current_term: 1 leader_uuid: "3402a9aa258142149fc95120884519c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } health_report { overall_health: HEALTHY } } }
I20260924 12:48:53.137650 14524 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cd86ffc0f7c141cabdfe32601456adbd"
dest_uuid: "3402a9aa258142149fc95120884519c0"
from {username='slave'} at 127.0.0.1:60182
I20260924 12:48:53.137846 14524 raft_consensus.cc:482] T cd86ffc0f7c141cabdfe32601456adbd P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 12:48:53.137941 14524 raft_consensus.cc:3037] T cd86ffc0f7c141cabdfe32601456adbd P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:53.139868 14524 raft_consensus.cc:504] T cd86ffc0f7c141cabdfe32601456adbd P 3402a9aa258142149fc95120884519c0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:53.140132 14524 leader_election.cc:302] T cd86ffc0f7c141cabdfe32601456adbd P 3402a9aa258142149fc95120884519c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3402a9aa258142149fc95120884519c0; no voters:
I20260924 12:48:53.140328 14524 leader_election.cc:288] T cd86ffc0f7c141cabdfe32601456adbd P 3402a9aa258142149fc95120884519c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:48:53.140432 14590 raft_consensus.cc:2781] T cd86ffc0f7c141cabdfe32601456adbd P 3402a9aa258142149fc95120884519c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:53.140586 14590 raft_consensus.cc:686] T cd86ffc0f7c141cabdfe32601456adbd P 3402a9aa258142149fc95120884519c0 [term 1 LEADER]: Becoming Leader. State: Replica: 3402a9aa258142149fc95120884519c0, State: Running, Role: LEADER
I20260924 12:48:53.140880 14590 consensus_queue.cc:237] T cd86ffc0f7c141cabdfe32601456adbd P 3402a9aa258142149fc95120884519c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:53.143028 14383 catalog_manager.cc:5495] T cd86ffc0f7c141cabdfe32601456adbd P 3402a9aa258142149fc95120884519c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3402a9aa258142149fc95120884519c0 (127.5.77.193). New cstate: current_term: 1 leader_uuid: "3402a9aa258142149fc95120884519c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } health_report { overall_health: HEALTHY } } }
I20260924 12:48:53.151455 14524 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9b25405e75f94c069e182584aa5a987a"
dest_uuid: "3402a9aa258142149fc95120884519c0"
from {username='slave'} at 127.0.0.1:60182
I20260924 12:48:53.151652 14524 raft_consensus.cc:482] T 9b25405e75f94c069e182584aa5a987a P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 12:48:53.151836 14524 raft_consensus.cc:3037] T 9b25405e75f94c069e182584aa5a987a P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:53.153594 14524 raft_consensus.cc:504] T 9b25405e75f94c069e182584aa5a987a P 3402a9aa258142149fc95120884519c0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:53.153834 14524 leader_election.cc:302] T 9b25405e75f94c069e182584aa5a987a P 3402a9aa258142149fc95120884519c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3402a9aa258142149fc95120884519c0; no voters:
I20260924 12:48:53.154006 14524 leader_election.cc:288] T 9b25405e75f94c069e182584aa5a987a P 3402a9aa258142149fc95120884519c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:48:53.154126 14590 raft_consensus.cc:2781] T 9b25405e75f94c069e182584aa5a987a P 3402a9aa258142149fc95120884519c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:53.154304 14590 raft_consensus.cc:686] T 9b25405e75f94c069e182584aa5a987a P 3402a9aa258142149fc95120884519c0 [term 1 LEADER]: Becoming Leader. State: Replica: 3402a9aa258142149fc95120884519c0, State: Running, Role: LEADER
I20260924 12:48:53.154556 14590 consensus_queue.cc:237] T 9b25405e75f94c069e182584aa5a987a P 3402a9aa258142149fc95120884519c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:53.156793 14383 catalog_manager.cc:5495] T 9b25405e75f94c069e182584aa5a987a P 3402a9aa258142149fc95120884519c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3402a9aa258142149fc95120884519c0 (127.5.77.193). New cstate: current_term: 1 leader_uuid: "3402a9aa258142149fc95120884519c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } health_report { overall_health: HEALTHY } } }
I20260924 12:48:53.165841 14524 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "467fe78d496a40ddaced927da77f01a6"
dest_uuid: "3402a9aa258142149fc95120884519c0"
from {username='slave'} at 127.0.0.1:60182
I20260924 12:48:53.166024 14524 raft_consensus.cc:482] T 467fe78d496a40ddaced927da77f01a6 P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 12:48:53.166113 14524 raft_consensus.cc:3037] T 467fe78d496a40ddaced927da77f01a6 P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:53.167996 14524 raft_consensus.cc:504] T 467fe78d496a40ddaced927da77f01a6 P 3402a9aa258142149fc95120884519c0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:53.168226 14524 leader_election.cc:302] T 467fe78d496a40ddaced927da77f01a6 P 3402a9aa258142149fc95120884519c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3402a9aa258142149fc95120884519c0; no voters:
I20260924 12:48:53.168360 14524 leader_election.cc:288] T 467fe78d496a40ddaced927da77f01a6 P 3402a9aa258142149fc95120884519c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:48:53.168466 14590 raft_consensus.cc:2781] T 467fe78d496a40ddaced927da77f01a6 P 3402a9aa258142149fc95120884519c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:53.168722 14590 raft_consensus.cc:686] T 467fe78d496a40ddaced927da77f01a6 P 3402a9aa258142149fc95120884519c0 [term 1 LEADER]: Becoming Leader. State: Replica: 3402a9aa258142149fc95120884519c0, State: Running, Role: LEADER
I20260924 12:48:53.168978 14590 consensus_queue.cc:237] T 467fe78d496a40ddaced927da77f01a6 P 3402a9aa258142149fc95120884519c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:53.171095 14383 catalog_manager.cc:5495] T 467fe78d496a40ddaced927da77f01a6 P 3402a9aa258142149fc95120884519c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3402a9aa258142149fc95120884519c0 (127.5.77.193). New cstate: current_term: 1 leader_uuid: "3402a9aa258142149fc95120884519c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } health_report { overall_health: HEALTHY } } }
I20260924 12:48:53.180228 14524 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f391e251916144f9827ad1ec9a82d25e"
dest_uuid: "3402a9aa258142149fc95120884519c0"
from {username='slave'} at 127.0.0.1:60182
I20260924 12:48:53.180421 14524 raft_consensus.cc:482] T f391e251916144f9827ad1ec9a82d25e P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 12:48:53.180512 14524 raft_consensus.cc:3037] T f391e251916144f9827ad1ec9a82d25e P 3402a9aa258142149fc95120884519c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:53.182323 14524 raft_consensus.cc:504] T f391e251916144f9827ad1ec9a82d25e P 3402a9aa258142149fc95120884519c0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:53.182619 14524 leader_election.cc:302] T f391e251916144f9827ad1ec9a82d25e P 3402a9aa258142149fc95120884519c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3402a9aa258142149fc95120884519c0; no voters:
I20260924 12:48:53.182790 14524 leader_election.cc:288] T f391e251916144f9827ad1ec9a82d25e P 3402a9aa258142149fc95120884519c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:48:53.182875 14590 raft_consensus.cc:2781] T f391e251916144f9827ad1ec9a82d25e P 3402a9aa258142149fc95120884519c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:53.183053 14590 raft_consensus.cc:686] T f391e251916144f9827ad1ec9a82d25e P 3402a9aa258142149fc95120884519c0 [term 1 LEADER]: Becoming Leader. State: Replica: 3402a9aa258142149fc95120884519c0, State: Running, Role: LEADER
I20260924 12:48:53.183274 14590 consensus_queue.cc:237] T f391e251916144f9827ad1ec9a82d25e P 3402a9aa258142149fc95120884519c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:53.185633 14383 catalog_manager.cc:5495] T f391e251916144f9827ad1ec9a82d25e P 3402a9aa258142149fc95120884519c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3402a9aa258142149fc95120884519c0 (127.5.77.193). New cstate: current_term: 1 leader_uuid: "3402a9aa258142149fc95120884519c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } health_report { overall_health: HEALTHY } } }
I20260924 12:48:53.194591 14502 tablet_service.cc:1514] Processing DeleteTablet for tablet 693d9014ea8b4162b5e9acae939e0d5d with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:60182
I20260924 12:48:53.195464 14614 tablet_replica.cc:330] T 693d9014ea8b4162b5e9acae939e0d5d P 3402a9aa258142149fc95120884519c0: stopping tablet replica
I20260924 12:48:53.195917 14614 raft_consensus.cc:2229] T 693d9014ea8b4162b5e9acae939e0d5d P 3402a9aa258142149fc95120884519c0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:48:53.196265 14614 raft_consensus.cc:2258] T 693d9014ea8b4162b5e9acae939e0d5d P 3402a9aa258142149fc95120884519c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:48:53.197737 14614 ts_tablet_manager.cc:1905] T 693d9014ea8b4162b5e9acae939e0d5d P 3402a9aa258142149fc95120884519c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 12:48:53.202267 14614 ts_tablet_manager.cc:1918] T 693d9014ea8b4162b5e9acae939e0d5d P 3402a9aa258142149fc95120884519c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260924 12:48:53.202395 14614 log.cc:1196] T 693d9014ea8b4162b5e9acae939e0d5d P 3402a9aa258142149fc95120884519c0: Deleting WAL directory at /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/wals/693d9014ea8b4162b5e9acae939e0d5d
I20260924 12:48:53.212206 14502 tablet_service.cc:1514] Processing DeleteTablet for tablet 83425546b6d2468892765374a54f21ef with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:60182
I20260924 12:48:53.212517 14614 tablet_replica.cc:330] T 83425546b6d2468892765374a54f21ef P 3402a9aa258142149fc95120884519c0: stopping tablet replica
I20260924 12:48:53.212706 14614 raft_consensus.cc:2229] T 83425546b6d2468892765374a54f21ef P 3402a9aa258142149fc95120884519c0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:48:53.212891 14614 raft_consensus.cc:2258] T 83425546b6d2468892765374a54f21ef P 3402a9aa258142149fc95120884519c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:48:53.213629 14614 ts_tablet_manager.cc:1905] T 83425546b6d2468892765374a54f21ef P 3402a9aa258142149fc95120884519c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 12:48:53.217617 14614 ts_tablet_manager.cc:1918] T 83425546b6d2468892765374a54f21ef P 3402a9aa258142149fc95120884519c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260924 12:48:53.217716 14614 log.cc:1196] T 83425546b6d2468892765374a54f21ef P 3402a9aa258142149fc95120884519c0: Deleting WAL directory at /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/wals/83425546b6d2468892765374a54f21ef
I20260924 12:48:53.227471 14502 tablet_service.cc:1514] Processing DeleteTablet for tablet 7df1b7ec436041a19c28b65271ecfd4d with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:60182
I20260924 12:48:53.227831 14614 tablet_replica.cc:330] T 7df1b7ec436041a19c28b65271ecfd4d P 3402a9aa258142149fc95120884519c0: stopping tablet replica
I20260924 12:48:53.228017 14614 raft_consensus.cc:2229] T 7df1b7ec436041a19c28b65271ecfd4d P 3402a9aa258142149fc95120884519c0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:48:53.228224 14614 raft_consensus.cc:2258] T 7df1b7ec436041a19c28b65271ecfd4d P 3402a9aa258142149fc95120884519c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:48:53.229054 14614 ts_tablet_manager.cc:1905] T 7df1b7ec436041a19c28b65271ecfd4d P 3402a9aa258142149fc95120884519c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 12:48:53.233148 14614 ts_tablet_manager.cc:1918] T 7df1b7ec436041a19c28b65271ecfd4d P 3402a9aa258142149fc95120884519c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260924 12:48:53.233253 14614 log.cc:1196] T 7df1b7ec436041a19c28b65271ecfd4d P 3402a9aa258142149fc95120884519c0: Deleting WAL directory at /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/wals/7df1b7ec436041a19c28b65271ecfd4d
I20260924 12:48:53.242883 14502 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fa30b54787449159eb687f1d9a38500 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:60182
I20260924 12:48:53.243198 14614 tablet_replica.cc:330] T 8fa30b54787449159eb687f1d9a38500 P 3402a9aa258142149fc95120884519c0: stopping tablet replica
I20260924 12:48:53.243378 14614 raft_consensus.cc:2229] T 8fa30b54787449159eb687f1d9a38500 P 3402a9aa258142149fc95120884519c0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:48:53.243582 14614 raft_consensus.cc:2258] T 8fa30b54787449159eb687f1d9a38500 P 3402a9aa258142149fc95120884519c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:48:53.244377 14614 ts_tablet_manager.cc:1905] T 8fa30b54787449159eb687f1d9a38500 P 3402a9aa258142149fc95120884519c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 12:48:53.248382 14614 ts_tablet_manager.cc:1918] T 8fa30b54787449159eb687f1d9a38500 P 3402a9aa258142149fc95120884519c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260924 12:48:53.248489 14614 log.cc:1196] T 8fa30b54787449159eb687f1d9a38500 P 3402a9aa258142149fc95120884519c0: Deleting WAL directory at /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/wals/8fa30b54787449159eb687f1d9a38500
I20260924 12:48:53.257907 14502 tablet_service.cc:1514] Processing DeleteTablet for tablet 778dc068540e42cca8fbee526ee7b03f with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:60182
I20260924 12:48:53.258198 14614 tablet_replica.cc:330] T 778dc068540e42cca8fbee526ee7b03f P 3402a9aa258142149fc95120884519c0: stopping tablet replica
I20260924 12:48:53.258371 14614 raft_consensus.cc:2229] T 778dc068540e42cca8fbee526ee7b03f P 3402a9aa258142149fc95120884519c0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:48:53.258577 14614 raft_consensus.cc:2258] T 778dc068540e42cca8fbee526ee7b03f P 3402a9aa258142149fc95120884519c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:48:53.259371 14614 ts_tablet_manager.cc:1905] T 778dc068540e42cca8fbee526ee7b03f P 3402a9aa258142149fc95120884519c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 12:48:53.263509 14614 ts_tablet_manager.cc:1918] T 778dc068540e42cca8fbee526ee7b03f P 3402a9aa258142149fc95120884519c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260924 12:48:53.263622 14614 log.cc:1196] T 778dc068540e42cca8fbee526ee7b03f P 3402a9aa258142149fc95120884519c0: Deleting WAL directory at /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/wals/778dc068540e42cca8fbee526ee7b03f
I20260924 12:48:53.273276 14524 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "edd31a12eb714c3f9b473fc6e45be023"
dest_uuid: "3402a9aa258142149fc95120884519c0"
from {username='slave'} at 127.0.0.1:60182
I20260924 12:48:53.273558 14524 raft_consensus.cc:580] T edd31a12eb714c3f9b473fc6e45be023 P 3402a9aa258142149fc95120884519c0 [term 1 LEADER]: Received request to step down
I20260924 12:48:53.273716 14524 raft_consensus.cc:3032] T edd31a12eb714c3f9b473fc6e45be023 P 3402a9aa258142149fc95120884519c0 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 12:48:53.273800 14524 raft_consensus.cc:729] T edd31a12eb714c3f9b473fc6e45be023 P 3402a9aa258142149fc95120884519c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3402a9aa258142149fc95120884519c0, State: Running, Role: LEADER
I20260924 12:48:53.274084 14524 consensus_queue.cc:260] T edd31a12eb714c3f9b473fc6e45be023 P 3402a9aa258142149fc95120884519c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:53.274248 14524 raft_consensus.cc:3037] T edd31a12eb714c3f9b473fc6e45be023 P 3402a9aa258142149fc95120884519c0 [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:48:53.283919 14502 tablet_service.cc:1514] Processing DeleteTablet for tablet edd31a12eb714c3f9b473fc6e45be023 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:60182
I20260924 12:48:53.284217 14614 tablet_replica.cc:330] T edd31a12eb714c3f9b473fc6e45be023 P 3402a9aa258142149fc95120884519c0: stopping tablet replica
I20260924 12:48:53.284408 14614 raft_consensus.cc:2229] T edd31a12eb714c3f9b473fc6e45be023 P 3402a9aa258142149fc95120884519c0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260924 12:48:53.284602 14614 raft_consensus.cc:2258] T edd31a12eb714c3f9b473fc6e45be023 P 3402a9aa258142149fc95120884519c0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260924 12:48:53.285486 14614 ts_tablet_manager.cc:1905] T edd31a12eb714c3f9b473fc6e45be023 P 3402a9aa258142149fc95120884519c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 12:48:53.289497 14614 ts_tablet_manager.cc:1918] T edd31a12eb714c3f9b473fc6e45be023 P 3402a9aa258142149fc95120884519c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260924 12:48:53.289600 14614 log.cc:1196] T edd31a12eb714c3f9b473fc6e45be023 P 3402a9aa258142149fc95120884519c0: Deleting WAL directory at /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/wals/edd31a12eb714c3f9b473fc6e45be023
I20260924 12:48:53.299324 14524 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cd86ffc0f7c141cabdfe32601456adbd"
dest_uuid: "3402a9aa258142149fc95120884519c0"
from {username='slave'} at 127.0.0.1:60182
I20260924 12:48:53.299523 14524 raft_consensus.cc:580] T cd86ffc0f7c141cabdfe32601456adbd P 3402a9aa258142149fc95120884519c0 [term 1 LEADER]: Received request to step down
I20260924 12:48:53.299604 14524 raft_consensus.cc:3032] T cd86ffc0f7c141cabdfe32601456adbd P 3402a9aa258142149fc95120884519c0 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 12:48:53.299656 14524 raft_consensus.cc:729] T cd86ffc0f7c141cabdfe32601456adbd P 3402a9aa258142149fc95120884519c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3402a9aa258142149fc95120884519c0, State: Running, Role: LEADER
I20260924 12:48:53.299834 14524 consensus_queue.cc:260] T cd86ffc0f7c141cabdfe32601456adbd P 3402a9aa258142149fc95120884519c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:53.299964 14524 raft_consensus.cc:3037] T cd86ffc0f7c141cabdfe32601456adbd P 3402a9aa258142149fc95120884519c0 [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:48:53.309422 14502 tablet_service.cc:1514] Processing DeleteTablet for tablet cd86ffc0f7c141cabdfe32601456adbd with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:60182
I20260924 12:48:53.309710 14614 tablet_replica.cc:330] T cd86ffc0f7c141cabdfe32601456adbd P 3402a9aa258142149fc95120884519c0: stopping tablet replica
I20260924 12:48:53.309863 14614 raft_consensus.cc:2229] T cd86ffc0f7c141cabdfe32601456adbd P 3402a9aa258142149fc95120884519c0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260924 12:48:53.310029 14614 raft_consensus.cc:2258] T cd86ffc0f7c141cabdfe32601456adbd P 3402a9aa258142149fc95120884519c0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260924 12:48:53.310837 14614 ts_tablet_manager.cc:1905] T cd86ffc0f7c141cabdfe32601456adbd P 3402a9aa258142149fc95120884519c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 12:48:53.314965 14614 ts_tablet_manager.cc:1918] T cd86ffc0f7c141cabdfe32601456adbd P 3402a9aa258142149fc95120884519c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260924 12:48:53.315100 14614 log.cc:1196] T cd86ffc0f7c141cabdfe32601456adbd P 3402a9aa258142149fc95120884519c0: Deleting WAL directory at /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/wals/cd86ffc0f7c141cabdfe32601456adbd
I20260924 12:48:53.324566 14524 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9b25405e75f94c069e182584aa5a987a"
dest_uuid: "3402a9aa258142149fc95120884519c0"
from {username='slave'} at 127.0.0.1:60182
I20260924 12:48:53.324810 14524 raft_consensus.cc:580] T 9b25405e75f94c069e182584aa5a987a P 3402a9aa258142149fc95120884519c0 [term 1 LEADER]: Received request to step down
I20260924 12:48:53.324911 14524 raft_consensus.cc:3032] T 9b25405e75f94c069e182584aa5a987a P 3402a9aa258142149fc95120884519c0 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 12:48:53.324975 14524 raft_consensus.cc:729] T 9b25405e75f94c069e182584aa5a987a P 3402a9aa258142149fc95120884519c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3402a9aa258142149fc95120884519c0, State: Running, Role: LEADER
I20260924 12:48:53.325182 14524 consensus_queue.cc:260] T 9b25405e75f94c069e182584aa5a987a P 3402a9aa258142149fc95120884519c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:53.325325 14524 raft_consensus.cc:3037] T 9b25405e75f94c069e182584aa5a987a P 3402a9aa258142149fc95120884519c0 [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:48:53.334403 14502 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b25405e75f94c069e182584aa5a987a with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:60182
I20260924 12:48:53.334725 14614 tablet_replica.cc:330] T 9b25405e75f94c069e182584aa5a987a P 3402a9aa258142149fc95120884519c0: stopping tablet replica
I20260924 12:48:53.334882 14614 raft_consensus.cc:2229] T 9b25405e75f94c069e182584aa5a987a P 3402a9aa258142149fc95120884519c0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260924 12:48:53.335059 14614 raft_consensus.cc:2258] T 9b25405e75f94c069e182584aa5a987a P 3402a9aa258142149fc95120884519c0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260924 12:48:53.335853 14614 ts_tablet_manager.cc:1905] T 9b25405e75f94c069e182584aa5a987a P 3402a9aa258142149fc95120884519c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 12:48:53.340006 14614 ts_tablet_manager.cc:1918] T 9b25405e75f94c069e182584aa5a987a P 3402a9aa258142149fc95120884519c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260924 12:48:53.340155 14614 log.cc:1196] T 9b25405e75f94c069e182584aa5a987a P 3402a9aa258142149fc95120884519c0: Deleting WAL directory at /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/wals/9b25405e75f94c069e182584aa5a987a
I20260924 12:48:53.349799 14524 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "467fe78d496a40ddaced927da77f01a6"
dest_uuid: "3402a9aa258142149fc95120884519c0"
from {username='slave'} at 127.0.0.1:60182
I20260924 12:48:53.350003 14524 raft_consensus.cc:580] T 467fe78d496a40ddaced927da77f01a6 P 3402a9aa258142149fc95120884519c0 [term 1 LEADER]: Received request to step down
I20260924 12:48:53.350080 14524 raft_consensus.cc:3032] T 467fe78d496a40ddaced927da77f01a6 P 3402a9aa258142149fc95120884519c0 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 12:48:53.350126 14524 raft_consensus.cc:729] T 467fe78d496a40ddaced927da77f01a6 P 3402a9aa258142149fc95120884519c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3402a9aa258142149fc95120884519c0, State: Running, Role: LEADER
I20260924 12:48:53.350293 14524 consensus_queue.cc:260] T 467fe78d496a40ddaced927da77f01a6 P 3402a9aa258142149fc95120884519c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:53.350423 14524 raft_consensus.cc:3037] T 467fe78d496a40ddaced927da77f01a6 P 3402a9aa258142149fc95120884519c0 [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:48:53.359472 14502 tablet_service.cc:1514] Processing DeleteTablet for tablet 467fe78d496a40ddaced927da77f01a6 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:60182
I20260924 12:48:53.359740 14614 tablet_replica.cc:330] T 467fe78d496a40ddaced927da77f01a6 P 3402a9aa258142149fc95120884519c0: stopping tablet replica
I20260924 12:48:53.359882 14614 raft_consensus.cc:2229] T 467fe78d496a40ddaced927da77f01a6 P 3402a9aa258142149fc95120884519c0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260924 12:48:53.360046 14614 raft_consensus.cc:2258] T 467fe78d496a40ddaced927da77f01a6 P 3402a9aa258142149fc95120884519c0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260924 12:48:53.360831 14614 ts_tablet_manager.cc:1905] T 467fe78d496a40ddaced927da77f01a6 P 3402a9aa258142149fc95120884519c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 12:48:53.365043 14614 ts_tablet_manager.cc:1918] T 467fe78d496a40ddaced927da77f01a6 P 3402a9aa258142149fc95120884519c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260924 12:48:53.365142 14614 log.cc:1196] T 467fe78d496a40ddaced927da77f01a6 P 3402a9aa258142149fc95120884519c0: Deleting WAL directory at /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/wals/467fe78d496a40ddaced927da77f01a6
I20260924 12:48:53.374605 14524 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f391e251916144f9827ad1ec9a82d25e"
dest_uuid: "3402a9aa258142149fc95120884519c0"
from {username='slave'} at 127.0.0.1:60182
I20260924 12:48:53.374837 14524 raft_consensus.cc:580] T f391e251916144f9827ad1ec9a82d25e P 3402a9aa258142149fc95120884519c0 [term 1 LEADER]: Received request to step down
I20260924 12:48:53.374931 14524 raft_consensus.cc:3032] T f391e251916144f9827ad1ec9a82d25e P 3402a9aa258142149fc95120884519c0 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 12:48:53.374997 14524 raft_consensus.cc:729] T f391e251916144f9827ad1ec9a82d25e P 3402a9aa258142149fc95120884519c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3402a9aa258142149fc95120884519c0, State: Running, Role: LEADER
I20260924 12:48:53.375212 14524 consensus_queue.cc:260] T f391e251916144f9827ad1ec9a82d25e P 3402a9aa258142149fc95120884519c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3402a9aa258142149fc95120884519c0" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 42193 } }
I20260924 12:48:53.375350 14524 raft_consensus.cc:3037] T f391e251916144f9827ad1ec9a82d25e P 3402a9aa258142149fc95120884519c0 [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:48:53.384495 14502 tablet_service.cc:1514] Processing DeleteTablet for tablet f391e251916144f9827ad1ec9a82d25e with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:60182
I20260924 12:48:53.384802 14614 tablet_replica.cc:330] T f391e251916144f9827ad1ec9a82d25e P 3402a9aa258142149fc95120884519c0: stopping tablet replica
I20260924 12:48:53.384964 14614 raft_consensus.cc:2229] T f391e251916144f9827ad1ec9a82d25e P 3402a9aa258142149fc95120884519c0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260924 12:48:53.385145 14614 raft_consensus.cc:2258] T f391e251916144f9827ad1ec9a82d25e P 3402a9aa258142149fc95120884519c0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260924 12:48:53.385968 14614 ts_tablet_manager.cc:1905] T f391e251916144f9827ad1ec9a82d25e P 3402a9aa258142149fc95120884519c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 12:48:53.389923 14614 ts_tablet_manager.cc:1918] T f391e251916144f9827ad1ec9a82d25e P 3402a9aa258142149fc95120884519c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260924 12:48:53.390020 14614 log.cc:1196] T f391e251916144f9827ad1ec9a82d25e P 3402a9aa258142149fc95120884519c0: Deleting WAL directory at /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/wals/f391e251916144f9827ad1ec9a82d25e
I20260924 12:48:53.400597 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 14442
I20260924 12:48:53.407320 14564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:53.514026 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 14442
I20260924 12:48:53.529448 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 14352
I20260924 12:48:53.530551 14413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:53.633464 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 14352
2026-09-24T12:48:53Z chronyd exiting
[ OK ] NumReplicas/RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric/0 (1845 ms)
[ RUN ] NumReplicas/RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric/1
I20260924 12:48:53.667017 5431 test_util.cc:268] Using random seed: 95544498
I20260924 12:48:53.668573 5431 ts_itest-base.cc:112] Starting cluster with:
I20260924 12:48:53.668633 5431 ts_itest-base.cc:113] --------------
I20260924 12:48:53.668690 5431 ts_itest-base.cc:114] 3 tablet servers
I20260924 12:48:53.668735 5431 ts_itest-base.cc:115] 3 replicas per TS
I20260924 12:48:53.668792 5431 ts_itest-base.cc:116] --------------
2026-09-24T12:48:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:48:53Z Disabled control of system clock
I20260924 12:48:53.705018 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.77.254:46757
--webserver_interface=127.5.77.254
--webserver_port=0
--builtin_ntp_servers=127.5.77.212:37395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.77.254:46757
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20260924 12:48:54.017772 14625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:54.018074 14625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:54.018190 14625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:54.025357 14625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:48:54.025467 14625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:54.025534 14625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:48:54.025604 14625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:48:54.034605 14625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:37395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.5.77.254:46757
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.77.254:46757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.77.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:54.035880 14625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:54.037560 14625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:54.048470 14630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:54.048458 14633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:54.048591 14631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:54.048949 14625 server_base.cc:1034] running on GCE node
I20260924 12:48:54.049778 14625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:54.051143 14625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:54.052388 14625 hybrid_clock.cc:648] HybridClock initialized: now 1790254134052321 us; error 38 us; skew 500 ppm
I20260924 12:48:54.052733 14625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:54.054939 14625 webserver.cc:460] Webserver started at http://127.5.77.254:35329/ using document root <none> and password file <none>
I20260924 12:48:54.055601 14625 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:54.055717 14625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:54.055985 14625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:54.058878 14625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "627578167adc4c66a5892858229edb41"
format_stamp: "Formatted at 2026-09-24 12:48:54 on dist-test-slave-b8q3"
I20260924 12:48:54.059610 14625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "627578167adc4c66a5892858229edb41"
format_stamp: "Formatted at 2026-09-24 12:48:54 on dist-test-slave-b8q3"
I20260924 12:48:54.064682 14625 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 12:48:54.068216 14639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:54.069669 14625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 12:48:54.069839 14625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
uuid: "627578167adc4c66a5892858229edb41"
format_stamp: "Formatted at 2026-09-24 12:48:54 on dist-test-slave-b8q3"
I20260924 12:48:54.070012 14625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:54.085870 14625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:54.086767 14625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:54.087028 14625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:54.103600 14625 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.254:46757
I20260924 12:48:54.103623 14690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.254:46757 every 8 connection(s)
I20260924 12:48:54.104948 14625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260924 12:48:54.107326 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 14625
I20260924 12:48:54.107473 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260924 12:48:54.109858 14691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:54.120550 14691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 627578167adc4c66a5892858229edb41: Bootstrap starting.
I20260924 12:48:54.124076 14691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 627578167adc4c66a5892858229edb41: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:54.125159 14691 log.cc:824] T 00000000000000000000000000000000 P 627578167adc4c66a5892858229edb41: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:54.127811 14691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 627578167adc4c66a5892858229edb41: No bootstrap required, opened a new log
I20260924 12:48:54.134934 14691 raft_consensus.cc:348] T 00000000000000000000000000000000 P 627578167adc4c66a5892858229edb41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627578167adc4c66a5892858229edb41" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 46757 } }
I20260924 12:48:54.135313 14691 raft_consensus.cc:374] T 00000000000000000000000000000000 P 627578167adc4c66a5892858229edb41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:48:54.135412 14691 raft_consensus.cc:729] T 00000000000000000000000000000000 P 627578167adc4c66a5892858229edb41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 627578167adc4c66a5892858229edb41, State: Initialized, Role: FOLLOWER
I20260924 12:48:54.135972 14691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 627578167adc4c66a5892858229edb41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627578167adc4c66a5892858229edb41" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 46757 } }
I20260924 12:48:54.136184 14691 raft_consensus.cc:388] T 00000000000000000000000000000000 P 627578167adc4c66a5892858229edb41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:48:54.136314 14691 raft_consensus.cc:482] T 00000000000000000000000000000000 P 627578167adc4c66a5892858229edb41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:48:54.136462 14691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 627578167adc4c66a5892858229edb41 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:54.138540 14691 raft_consensus.cc:504] T 00000000000000000000000000000000 P 627578167adc4c66a5892858229edb41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627578167adc4c66a5892858229edb41" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 46757 } }
I20260924 12:48:54.139113 14691 leader_election.cc:302] T 00000000000000000000000000000000 P 627578167adc4c66a5892858229edb41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 627578167adc4c66a5892858229edb41; no voters:
I20260924 12:48:54.139701 14691 leader_election.cc:288] T 00000000000000000000000000000000 P 627578167adc4c66a5892858229edb41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:48:54.139811 14696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 627578167adc4c66a5892858229edb41 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:54.140447 14696 raft_consensus.cc:686] T 00000000000000000000000000000000 P 627578167adc4c66a5892858229edb41 [term 1 LEADER]: Becoming Leader. State: Replica: 627578167adc4c66a5892858229edb41, State: Running, Role: LEADER
I20260924 12:48:54.140954 14696 consensus_queue.cc:237] T 00000000000000000000000000000000 P 627578167adc4c66a5892858229edb41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627578167adc4c66a5892858229edb41" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 46757 } }
I20260924 12:48:54.141662 14691 sys_catalog.cc:564] T 00000000000000000000000000000000 P 627578167adc4c66a5892858229edb41 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:48:54.143610 14697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 627578167adc4c66a5892858229edb41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "627578167adc4c66a5892858229edb41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627578167adc4c66a5892858229edb41" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 46757 } } }
I20260924 12:48:54.144023 14697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 627578167adc4c66a5892858229edb41 [sys.catalog]: This master's current role is: LEADER
I20260924 12:48:54.143817 14698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 627578167adc4c66a5892858229edb41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 627578167adc4c66a5892858229edb41. Latest consensus state: current_term: 1 leader_uuid: "627578167adc4c66a5892858229edb41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627578167adc4c66a5892858229edb41" member_type: VOTER last_known_addr { host: "127.5.77.254" port: 46757 } } }
I20260924 12:48:54.144145 14698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 627578167adc4c66a5892858229edb41 [sys.catalog]: This master's current role is: LEADER
I20260924 12:48:54.145637 14704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:48:54.150444 14704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:48:54.158762 14704 catalog_manager.cc:1347] Generated new cluster ID: c401ddfe3f7f4d289d12751106f2fa2b
I20260924 12:48:54.158896 14704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:48:54.175069 14704 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:48:54.176367 14704 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:48:54.183560 14704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 627578167adc4c66a5892858229edb41: Generated new TSK 0
I20260924 12:48:54.184444 14704 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:48:54.196378 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.193:0
--local_ip_for_outbound_sockets=127.5.77.193
--webserver_interface=127.5.77.193
--webserver_port=0
--tserver_master_addrs=127.5.77.254:46757
--builtin_ntp_servers=127.5.77.212:37395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20260924 12:48:54.514346 14715 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 12:48:54.514708 14715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:54.514801 14715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:54.514892 14715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:54.522092 14715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:54.522254 14715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.193
I20260924 12:48:54.531383 14715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:37395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.77.193
--webserver_port=0
--tserver_master_addrs=127.5.77.254:46757
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.193
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:54.532682 14715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:54.534349 14715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:54.547273 14723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:54.547252 14721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:54.547226 14720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:54.547926 14715 server_base.cc:1034] running on GCE node
I20260924 12:48:54.548534 14715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:54.549471 14715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:54.550676 14715 hybrid_clock.cc:648] HybridClock initialized: now 1790254134550623 us; error 74 us; skew 500 ppm
I20260924 12:48:54.551019 14715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:54.553212 14715 webserver.cc:460] Webserver started at http://127.5.77.193:39801/ using document root <none> and password file <none>
I20260924 12:48:54.553886 14715 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:54.553974 14715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:54.554266 14715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:54.556823 14715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "16464c128b8649d6bafdf4f45f981031"
format_stamp: "Formatted at 2026-09-24 12:48:54 on dist-test-slave-b8q3"
I20260924 12:48:54.557521 14715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "16464c128b8649d6bafdf4f45f981031"
format_stamp: "Formatted at 2026-09-24 12:48:54 on dist-test-slave-b8q3"
I20260924 12:48:54.562753 14715 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 12:48:54.566150 14729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:54.567587 14715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:48:54.567739 14715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "16464c128b8649d6bafdf4f45f981031"
format_stamp: "Formatted at 2026-09-24 12:48:54 on dist-test-slave-b8q3"
I20260924 12:48:54.567937 14715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:54.592443 14715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:54.593287 14715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:54.593566 14715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:54.594504 14715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:54.595932 14715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:48:54.596014 14715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:54.596117 14715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:48:54.596196 14715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:54.624327 14715 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.193:37587
I20260924 12:48:54.624395 14841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.193:37587 every 8 connection(s)
I20260924 12:48:54.625551 14715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260924 12:48:54.633286 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 14715
I20260924 12:48:54.633463 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260924 12:48:54.636646 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.194:0
--local_ip_for_outbound_sockets=127.5.77.194
--webserver_interface=127.5.77.194
--webserver_port=0
--tserver_master_addrs=127.5.77.254:46757
--builtin_ntp_servers=127.5.77.212:37395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260924 12:48:54.640587 14842 heartbeater.cc:347] Connected to a master server at 127.5.77.254:46757
I20260924 12:48:54.640875 14842 heartbeater.cc:464] Registering TS with master...
I20260924 12:48:54.641615 14842 heartbeater.cc:511] Master 127.5.77.254:46757 requested a full tablet report, sending...
I20260924 12:48:54.643573 14656 ts_manager.cc:194] Registered new tserver with Master: 16464c128b8649d6bafdf4f45f981031 (127.5.77.193:37587)
I20260924 12:48:54.645236 14656 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.193:49177
W20260924 12:48:54.955894 14846 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 12:48:54.956188 14846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:54.956254 14846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:54.956321 14846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:54.963604 14846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:54.963765 14846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.194
I20260924 12:48:54.973127 14846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:37395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.77.194
--webserver_port=0
--tserver_master_addrs=127.5.77.254:46757
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.194
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:54.974519 14846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:54.976322 14846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:54.989120 14851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:54.989140 14854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:54.989077 14852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:54.989859 14846 server_base.cc:1034] running on GCE node
I20260924 12:48:54.990486 14846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:54.991441 14846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:54.992669 14846 hybrid_clock.cc:648] HybridClock initialized: now 1790254134992581 us; error 49 us; skew 500 ppm
I20260924 12:48:54.992978 14846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:54.995213 14846 webserver.cc:460] Webserver started at http://127.5.77.194:40353/ using document root <none> and password file <none>
I20260924 12:48:54.995887 14846 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:54.995977 14846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:54.996258 14846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:54.998843 14846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "07ae11c0def5467ea650240db0071d78"
format_stamp: "Formatted at 2026-09-24 12:48:54 on dist-test-slave-b8q3"
I20260924 12:48:54.999562 14846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "07ae11c0def5467ea650240db0071d78"
format_stamp: "Formatted at 2026-09-24 12:48:54 on dist-test-slave-b8q3"
I20260924 12:48:55.004565 14846 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 12:48:55.007915 14860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:55.009284 14846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 12:48:55.009418 14846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "07ae11c0def5467ea650240db0071d78"
format_stamp: "Formatted at 2026-09-24 12:48:54 on dist-test-slave-b8q3"
I20260924 12:48:55.009603 14846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:55.036425 14846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:55.037290 14846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:55.037568 14846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:55.038550 14846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:55.040020 14846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:48:55.040104 14846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:55.040208 14846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:48:55.040272 14846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:55.069545 14846 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.194:34953
I20260924 12:48:55.069603 14972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.194:34953 every 8 connection(s)
I20260924 12:48:55.070879 14846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260924 12:48:55.071233 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 14846
I20260924 12:48:55.071370 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260924 12:48:55.075202 5431 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL2kveR/build/asan/bin/kudu
/tmp/dist-test-taskL2kveR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.77.195:0
--local_ip_for_outbound_sockets=127.5.77.195
--webserver_interface=127.5.77.195
--webserver_port=0
--tserver_master_addrs=127.5.77.254:46757
--builtin_ntp_servers=127.5.77.212:37395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260924 12:48:55.087173 14973 heartbeater.cc:347] Connected to a master server at 127.5.77.254:46757
I20260924 12:48:55.087469 14973 heartbeater.cc:464] Registering TS with master...
I20260924 12:48:55.088176 14973 heartbeater.cc:511] Master 127.5.77.254:46757 requested a full tablet report, sending...
I20260924 12:48:55.089555 14656 ts_manager.cc:194] Registered new tserver with Master: 07ae11c0def5467ea650240db0071d78 (127.5.77.194:34953)
I20260924 12:48:55.090588 14656 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.194:33641
W20260924 12:48:55.394855 14976 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 12:48:55.395145 14976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:55.395210 14976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:55.395310 14976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:55.402491 14976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:55.402652 14976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.77.195
I20260924 12:48:55.411621 14976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.77.212:37395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.77.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.77.195
--webserver_port=0
--tserver_master_addrs=127.5.77.254:46757
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.77.195
--log_dir=/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:55.412926 14976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:55.414719 14976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:55.427345 14982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:55.427403 14981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:55.427372 14984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:55.427933 14976 server_base.cc:1034] running on GCE node
I20260924 12:48:55.428577 14976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:55.429533 14976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:55.430750 14976 hybrid_clock.cc:648] HybridClock initialized: now 1790254135430689 us; error 52 us; skew 500 ppm
I20260924 12:48:55.431038 14976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:48:55.433266 14976 webserver.cc:460] Webserver started at http://127.5.77.195:43761/ using document root <none> and password file <none>
I20260924 12:48:55.433961 14976 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:55.434046 14976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:55.434298 14976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:48:55.436895 14976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7"
format_stamp: "Formatted at 2026-09-24 12:48:55 on dist-test-slave-b8q3"
I20260924 12:48:55.437582 14976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7"
format_stamp: "Formatted at 2026-09-24 12:48:55 on dist-test-slave-b8q3"
I20260924 12:48:55.442828 14976 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 12:48:55.446187 14990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:55.447703 14976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:48:55.447844 14976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7"
format_stamp: "Formatted at 2026-09-24 12:48:55 on dist-test-slave-b8q3"
I20260924 12:48:55.448033 14976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:55.467062 14976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:55.467952 14976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:55.468231 14976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:55.469206 14976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:55.470641 14976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:48:55.470727 14976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:55.470825 14976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:48:55.470883 14976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:55.504290 14976 rpc_server.cc:307] RPC server started. Bound to: 127.5.77.195:38363
I20260924 12:48:55.504546 15102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.77.195:38363 every 8 connection(s)
I20260924 12:48:55.505811 14976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260924 12:48:55.513264 5431 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL2kveR/build/asan/bin/kudu as pid 14976
I20260924 12:48:55.513437 5431 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260924 12:48:55.523228 15103 heartbeater.cc:347] Connected to a master server at 127.5.77.254:46757
I20260924 12:48:55.523576 15103 heartbeater.cc:464] Registering TS with master...
I20260924 12:48:55.524524 15103 heartbeater.cc:511] Master 127.5.77.254:46757 requested a full tablet report, sending...
I20260924 12:48:55.527330 14656 ts_manager.cc:194] Registered new tserver with Master: af5894d0b4fe4f3ea1ebb39472de0eb7 (127.5.77.195:38363)
I20260924 12:48:55.528102 14656 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.77.195:45699
I20260924 12:48:55.530608 5431 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:48:55.547210 5431 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:48:55.547417 5431 test_util.cc:268] Using random seed: 97424906
I20260924 12:48:55.565730 14656 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42620:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:48:55.567874 14656 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:48:55.608953 14777 tablet_service.cc:1467] Processing CreateTablet for tablet e9f03002a94945c692b7b26f332d95e8 (DEFAULT_TABLE table=test-workload [id=15a4deaa4d1f4c438be5a4339d3a44f7]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260924 12:48:55.609373 15038 tablet_service.cc:1467] Processing CreateTablet for tablet e9f03002a94945c692b7b26f332d95e8 (DEFAULT_TABLE table=test-workload [id=15a4deaa4d1f4c438be5a4339d3a44f7]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260924 12:48:55.609629 14905 tablet_service.cc:1467] Processing CreateTablet for tablet 3dd93acfeb704bc6b84977a1f6946aa3 (DEFAULT_TABLE table=test-workload [id=15a4deaa4d1f4c438be5a4339d3a44f7]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260924 12:48:55.610710 15035 tablet_service.cc:1467] Processing CreateTablet for tablet 3dd93acfeb704bc6b84977a1f6946aa3 (DEFAULT_TABLE table=test-workload [id=15a4deaa4d1f4c438be5a4339d3a44f7]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260924 12:48:55.610905 14777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9f03002a94945c692b7b26f332d95e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:55.611090 15033 tablet_service.cc:1467] Processing CreateTablet for tablet d011c630724442dfab74007165a31164 (DEFAULT_TABLE table=test-workload [id=15a4deaa4d1f4c438be5a4339d3a44f7]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260924 12:48:55.611142 14899 tablet_service.cc:1467] Processing CreateTablet for tablet 5c66212111554f04893157d46ea41a71 (DEFAULT_TABLE table=test-workload [id=15a4deaa4d1f4c438be5a4339d3a44f7]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260924 12:48:55.611418 14905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dd93acfeb704bc6b84977a1f6946aa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:55.611430 14776 tablet_service.cc:1467] Processing CreateTablet for tablet 2a500eb27ec84386ac2e26638a9c4221 (DEFAULT_TABLE table=test-workload [id=15a4deaa4d1f4c438be5a4339d3a44f7]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260924 12:48:55.611878 14776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a500eb27ec84386ac2e26638a9c4221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:55.611963 14900 tablet_service.cc:1467] Processing CreateTablet for tablet d396cddfe7674886b5ba7519d8495bf3 (DEFAULT_TABLE table=test-workload [id=15a4deaa4d1f4c438be5a4339d3a44f7]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260924 12:48:55.612453 14900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d396cddfe7674886b5ba7519d8495bf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:55.613040 15036 tablet_service.cc:1467] Processing CreateTablet for tablet 523d9fb810d440d5ba1b16fd3da57602 (DEFAULT_TABLE table=test-workload [id=15a4deaa4d1f4c438be5a4339d3a44f7]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260924 12:48:55.613493 15034 tablet_service.cc:1467] Processing CreateTablet for tablet 75c3b8a29aa1489da08a21b6cf353bc6 (DEFAULT_TABLE table=test-workload [id=15a4deaa4d1f4c438be5a4339d3a44f7]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260924 12:48:55.614564 15030 tablet_service.cc:1467] Processing CreateTablet for tablet d396cddfe7674886b5ba7519d8495bf3 (DEFAULT_TABLE table=test-workload [id=15a4deaa4d1f4c438be5a4339d3a44f7]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260924 12:48:55.614622 15029 tablet_service.cc:1467] Processing CreateTablet for tablet 5c66212111554f04893157d46ea41a71 (DEFAULT_TABLE table=test-workload [id=15a4deaa4d1f4c438be5a4339d3a44f7]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260924 12:48:55.614940 14901 tablet_service.cc:1467] Processing CreateTablet for tablet 6e2027d642db4900bea24b47baf00757 (DEFAULT_TABLE table=test-workload [id=15a4deaa4d1f4c438be5a4339d3a44f7]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260924 12:48:55.615007 15037 tablet_service.cc:1467] Processing CreateTablet for tablet 2a500eb27ec84386ac2e26638a9c4221 (DEFAULT_TABLE table=test-workload [id=15a4deaa4d1f4c438be5a4339d3a44f7]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260924 12:48:55.615288 14901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e2027d642db4900bea24b47baf00757. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:55.615262 15032 tablet_service.cc:1467] Processing CreateTablet for tablet 9723e825f07d48c9bc9fd927dd8ce48e (DEFAULT_TABLE table=test-workload [id=15a4deaa4d1f4c438be5a4339d3a44f7]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260924 12:48:55.615442 15031 tablet_service.cc:1467] Processing CreateTablet for tablet 6e2027d642db4900bea24b47baf00757 (DEFAULT_TABLE table=test-workload [id=15a4deaa4d1f4c438be5a4339d3a44f7]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260924 12:48:55.616209 14903 tablet_service.cc:1467] Processing CreateTablet for tablet d011c630724442dfab74007165a31164 (DEFAULT_TABLE table=test-workload [id=15a4deaa4d1f4c438be5a4339d3a44f7]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260924 12:48:55.616297 14904 tablet_service.cc:1467] Processing CreateTablet for tablet 75c3b8a29aa1489da08a21b6cf353bc6 (DEFAULT_TABLE table=test-workload [id=15a4deaa4d1f4c438be5a4339d3a44f7]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260924 12:48:55.616410 14906 tablet_service.cc:1467] Processing CreateTablet for tablet 523d9fb810d440d5ba1b16fd3da57602 (DEFAULT_TABLE table=test-workload [id=15a4deaa4d1f4c438be5a4339d3a44f7]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260924 12:48:55.616688 14903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d011c630724442dfab74007165a31164. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:55.616952 14904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75c3b8a29aa1489da08a21b6cf353bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:55.616997 14907 tablet_service.cc:1467] Processing CreateTablet for tablet 2a500eb27ec84386ac2e26638a9c4221 (DEFAULT_TABLE table=test-workload [id=15a4deaa4d1f4c438be5a4339d3a44f7]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260924 12:48:55.617285 14902 tablet_service.cc:1467] Processing CreateTablet for tablet 9723e825f07d48c9bc9fd927dd8ce48e (DEFAULT_TABLE table=test-workload [id=15a4deaa4d1f4c438be5a4339d3a44f7]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260924 12:48:55.617470 14907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a500eb27ec84386ac2e26638a9c4221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:55.617697 14902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9723e825f07d48c9bc9fd927dd8ce48e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:55.618237 14906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 523d9fb810d440d5ba1b16fd3da57602. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:55.618618 14899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c66212111554f04893157d46ea41a71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:55.610060 14908 tablet_service.cc:1467] Processing CreateTablet for tablet e9f03002a94945c692b7b26f332d95e8 (DEFAULT_TABLE table=test-workload [id=15a4deaa4d1f4c438be5a4339d3a44f7]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260924 12:48:55.619627 14908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9f03002a94945c692b7b26f332d95e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:55.623324 15038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9f03002a94945c692b7b26f332d95e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:55.626045 15035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dd93acfeb704bc6b84977a1f6946aa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:55.626971 15032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9723e825f07d48c9bc9fd927dd8ce48e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:55.627731 15037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a500eb27ec84386ac2e26638a9c4221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:55.628465 15029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c66212111554f04893157d46ea41a71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:55.629174 15031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e2027d642db4900bea24b47baf00757. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:55.630141 15033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d011c630724442dfab74007165a31164. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:55.630801 15034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75c3b8a29aa1489da08a21b6cf353bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:55.631451 15036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 523d9fb810d440d5ba1b16fd3da57602. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:55.632083 15030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d396cddfe7674886b5ba7519d8495bf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:55.639410 14775 tablet_service.cc:1467] Processing CreateTablet for tablet 523d9fb810d440d5ba1b16fd3da57602 (DEFAULT_TABLE table=test-workload [id=15a4deaa4d1f4c438be5a4339d3a44f7]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260924 12:48:55.639474 14774 tablet_service.cc:1467] Processing CreateTablet for tablet 3dd93acfeb704bc6b84977a1f6946aa3 (DEFAULT_TABLE table=test-workload [id=15a4deaa4d1f4c438be5a4339d3a44f7]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260924 12:48:55.639663 14771 tablet_service.cc:1467] Processing CreateTablet for tablet 9723e825f07d48c9bc9fd927dd8ce48e (DEFAULT_TABLE table=test-workload [id=15a4deaa4d1f4c438be5a4339d3a44f7]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260924 12:48:55.639983 14774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dd93acfeb704bc6b84977a1f6946aa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:55.640223 14771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9723e825f07d48c9bc9fd927dd8ce48e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:55.640156 14773 tablet_service.cc:1467] Processing CreateTablet for tablet 75c3b8a29aa1489da08a21b6cf353bc6 (DEFAULT_TABLE table=test-workload [id=15a4deaa4d1f4c438be5a4339d3a44f7]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260924 12:48:55.640894 14773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75c3b8a29aa1489da08a21b6cf353bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:55.640868 14772 tablet_service.cc:1467] Processing CreateTablet for tablet d011c630724442dfab74007165a31164 (DEFAULT_TABLE table=test-workload [id=15a4deaa4d1f4c438be5a4339d3a44f7]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260924 12:48:55.641242 14772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d011c630724442dfab74007165a31164. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:55.641527 14775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 523d9fb810d440d5ba1b16fd3da57602. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:55.648133 14842 heartbeater.cc:503] Master 127.5.77.254:46757 was elected leader, sending a full tablet report...
I20260924 12:48:55.680305 14773 tablet_service.cc:1467] Processing CreateTablet for tablet 6e2027d642db4900bea24b47baf00757 (DEFAULT_TABLE table=test-workload [id=15a4deaa4d1f4c438be5a4339d3a44f7]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260924 12:48:55.680402 15126 tablet_bootstrap.cc:492] T 6e2027d642db4900bea24b47baf00757 P 07ae11c0def5467ea650240db0071d78: Bootstrap starting.
I20260924 12:48:55.680701 14773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e2027d642db4900bea24b47baf00757. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:55.682804 14771 tablet_service.cc:1467] Processing CreateTablet for tablet d396cddfe7674886b5ba7519d8495bf3 (DEFAULT_TABLE table=test-workload [id=15a4deaa4d1f4c438be5a4339d3a44f7]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260924 12:48:55.683130 14771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d396cddfe7674886b5ba7519d8495bf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:55.684055 15126 tablet_bootstrap.cc:654] T 6e2027d642db4900bea24b47baf00757 P 07ae11c0def5467ea650240db0071d78: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:55.685339 15126 log.cc:824] T 6e2027d642db4900bea24b47baf00757 P 07ae11c0def5467ea650240db0071d78: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:55.686772 14775 tablet_service.cc:1467] Processing CreateTablet for tablet 5c66212111554f04893157d46ea41a71 (DEFAULT_TABLE table=test-workload [id=15a4deaa4d1f4c438be5a4339d3a44f7]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260924 12:48:55.687093 14775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c66212111554f04893157d46ea41a71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:55.691360 15126 tablet_bootstrap.cc:492] T 6e2027d642db4900bea24b47baf00757 P 07ae11c0def5467ea650240db0071d78: No bootstrap required, opened a new log
I20260924 12:48:55.691670 15126 ts_tablet_manager.cc:1397] T 6e2027d642db4900bea24b47baf00757 P 07ae11c0def5467ea650240db0071d78: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260924 12:48:55.701715 15127 tablet_bootstrap.cc:492] T d011c630724442dfab74007165a31164 P 16464c128b8649d6bafdf4f45f981031: Bootstrap starting.
I20260924 12:48:55.701862 15126 raft_consensus.cc:348] T 6e2027d642db4900bea24b47baf00757 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.702345 15126 raft_consensus.cc:729] T 6e2027d642db4900bea24b47baf00757 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae11c0def5467ea650240db0071d78, State: Initialized, Role: FOLLOWER
I20260924 12:48:55.702844 15126 consensus_queue.cc:260] T 6e2027d642db4900bea24b47baf00757 P 07ae11c0def5467ea650240db0071d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.706527 15129 tablet_bootstrap.cc:492] T d011c630724442dfab74007165a31164 P af5894d0b4fe4f3ea1ebb39472de0eb7: Bootstrap starting.
I20260924 12:48:55.707115 14973 heartbeater.cc:503] Master 127.5.77.254:46757 was elected leader, sending a full tablet report...
I20260924 12:48:55.707535 15127 tablet_bootstrap.cc:654] T d011c630724442dfab74007165a31164 P 16464c128b8649d6bafdf4f45f981031: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:55.707963 15126 ts_tablet_manager.cc:1428] T 6e2027d642db4900bea24b47baf00757 P 07ae11c0def5467ea650240db0071d78: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
I20260924 12:48:55.708460 15126 tablet_bootstrap.cc:492] T 75c3b8a29aa1489da08a21b6cf353bc6 P 07ae11c0def5467ea650240db0071d78: Bootstrap starting.
I20260924 12:48:55.710103 15126 tablet_bootstrap.cc:654] T 75c3b8a29aa1489da08a21b6cf353bc6 P 07ae11c0def5467ea650240db0071d78: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:55.710142 15129 tablet_bootstrap.cc:654] T d011c630724442dfab74007165a31164 P af5894d0b4fe4f3ea1ebb39472de0eb7: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:55.710215 15127 log.cc:824] T d011c630724442dfab74007165a31164 P 16464c128b8649d6bafdf4f45f981031: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:55.711377 15129 log.cc:824] T d011c630724442dfab74007165a31164 P af5894d0b4fe4f3ea1ebb39472de0eb7: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:55.715725 15126 tablet_bootstrap.cc:492] T 75c3b8a29aa1489da08a21b6cf353bc6 P 07ae11c0def5467ea650240db0071d78: No bootstrap required, opened a new log
I20260924 12:48:55.715909 15126 ts_tablet_manager.cc:1397] T 75c3b8a29aa1489da08a21b6cf353bc6 P 07ae11c0def5467ea650240db0071d78: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260924 12:48:55.716935 15127 tablet_bootstrap.cc:492] T d011c630724442dfab74007165a31164 P 16464c128b8649d6bafdf4f45f981031: No bootstrap required, opened a new log
I20260924 12:48:55.716801 15126 raft_consensus.cc:348] T 75c3b8a29aa1489da08a21b6cf353bc6 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.717064 15126 raft_consensus.cc:729] T 75c3b8a29aa1489da08a21b6cf353bc6 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae11c0def5467ea650240db0071d78, State: Initialized, Role: FOLLOWER
I20260924 12:48:55.717232 15127 ts_tablet_manager.cc:1397] T d011c630724442dfab74007165a31164 P 16464c128b8649d6bafdf4f45f981031: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.000s
I20260924 12:48:55.717296 15126 consensus_queue.cc:260] T 75c3b8a29aa1489da08a21b6cf353bc6 P 07ae11c0def5467ea650240db0071d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.717960 15129 tablet_bootstrap.cc:492] T d011c630724442dfab74007165a31164 P af5894d0b4fe4f3ea1ebb39472de0eb7: No bootstrap required, opened a new log
I20260924 12:48:55.718022 15126 ts_tablet_manager.cc:1428] T 75c3b8a29aa1489da08a21b6cf353bc6 P 07ae11c0def5467ea650240db0071d78: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 12:48:55.718261 15126 tablet_bootstrap.cc:492] T 9723e825f07d48c9bc9fd927dd8ce48e P 07ae11c0def5467ea650240db0071d78: Bootstrap starting.
I20260924 12:48:55.718367 15129 ts_tablet_manager.cc:1397] T d011c630724442dfab74007165a31164 P af5894d0b4fe4f3ea1ebb39472de0eb7: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260924 12:48:55.719976 15126 tablet_bootstrap.cc:654] T 9723e825f07d48c9bc9fd927dd8ce48e P 07ae11c0def5467ea650240db0071d78: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:55.725771 15126 tablet_bootstrap.cc:492] T 9723e825f07d48c9bc9fd927dd8ce48e P 07ae11c0def5467ea650240db0071d78: No bootstrap required, opened a new log
I20260924 12:48:55.725986 15126 ts_tablet_manager.cc:1397] T 9723e825f07d48c9bc9fd927dd8ce48e P 07ae11c0def5467ea650240db0071d78: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 12:48:55.726737 15126 raft_consensus.cc:348] T 9723e825f07d48c9bc9fd927dd8ce48e P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.726997 15126 raft_consensus.cc:729] T 9723e825f07d48c9bc9fd927dd8ce48e P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae11c0def5467ea650240db0071d78, State: Initialized, Role: FOLLOWER
I20260924 12:48:55.726758 15127 raft_consensus.cc:348] T d011c630724442dfab74007165a31164 P 16464c128b8649d6bafdf4f45f981031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.727169 15127 raft_consensus.cc:729] T d011c630724442dfab74007165a31164 P 16464c128b8649d6bafdf4f45f981031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16464c128b8649d6bafdf4f45f981031, State: Initialized, Role: FOLLOWER
I20260924 12:48:55.727185 15126 consensus_queue.cc:260] T 9723e825f07d48c9bc9fd927dd8ce48e P 07ae11c0def5467ea650240db0071d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.727767 15126 ts_tablet_manager.cc:1428] T 9723e825f07d48c9bc9fd927dd8ce48e P 07ae11c0def5467ea650240db0071d78: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 12:48:55.727645 15127 consensus_queue.cc:260] T d011c630724442dfab74007165a31164 P 16464c128b8649d6bafdf4f45f981031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.728071 15126 tablet_bootstrap.cc:492] T d011c630724442dfab74007165a31164 P 07ae11c0def5467ea650240db0071d78: Bootstrap starting.
I20260924 12:48:55.729115 15129 raft_consensus.cc:348] T d011c630724442dfab74007165a31164 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.729578 15129 raft_consensus.cc:729] T d011c630724442dfab74007165a31164 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af5894d0b4fe4f3ea1ebb39472de0eb7, State: Initialized, Role: FOLLOWER
I20260924 12:48:55.729743 15126 tablet_bootstrap.cc:654] T d011c630724442dfab74007165a31164 P 07ae11c0def5467ea650240db0071d78: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:55.730063 15129 consensus_queue.cc:260] T d011c630724442dfab74007165a31164 P af5894d0b4fe4f3ea1ebb39472de0eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.736891 15127 ts_tablet_manager.cc:1428] T d011c630724442dfab74007165a31164 P 16464c128b8649d6bafdf4f45f981031: Time spent starting tablet: real 0.020s user 0.020s sys 0.000s
I20260924 12:48:55.737303 15127 tablet_bootstrap.cc:492] T 523d9fb810d440d5ba1b16fd3da57602 P 16464c128b8649d6bafdf4f45f981031: Bootstrap starting.
I20260924 12:48:55.737496 15129 ts_tablet_manager.cc:1428] T d011c630724442dfab74007165a31164 P af5894d0b4fe4f3ea1ebb39472de0eb7: Time spent starting tablet: real 0.019s user 0.013s sys 0.005s
I20260924 12:48:55.738111 15103 heartbeater.cc:503] Master 127.5.77.254:46757 was elected leader, sending a full tablet report...
I20260924 12:48:55.738176 15129 tablet_bootstrap.cc:492] T d396cddfe7674886b5ba7519d8495bf3 P af5894d0b4fe4f3ea1ebb39472de0eb7: Bootstrap starting.
I20260924 12:48:55.738979 15127 tablet_bootstrap.cc:654] T 523d9fb810d440d5ba1b16fd3da57602 P 16464c128b8649d6bafdf4f45f981031: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:55.739899 15129 tablet_bootstrap.cc:654] T d396cddfe7674886b5ba7519d8495bf3 P af5894d0b4fe4f3ea1ebb39472de0eb7: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:55.742857 15129 tablet_bootstrap.cc:492] T d396cddfe7674886b5ba7519d8495bf3 P af5894d0b4fe4f3ea1ebb39472de0eb7: No bootstrap required, opened a new log
I20260924 12:48:55.742975 15127 tablet_bootstrap.cc:492] T 523d9fb810d440d5ba1b16fd3da57602 P 16464c128b8649d6bafdf4f45f981031: No bootstrap required, opened a new log
I20260924 12:48:55.743083 15129 ts_tablet_manager.cc:1397] T d396cddfe7674886b5ba7519d8495bf3 P af5894d0b4fe4f3ea1ebb39472de0eb7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 12:48:55.743209 15127 ts_tablet_manager.cc:1397] T 523d9fb810d440d5ba1b16fd3da57602 P 16464c128b8649d6bafdf4f45f981031: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 12:48:55.744068 15129 raft_consensus.cc:348] T d396cddfe7674886b5ba7519d8495bf3 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.744107 15127 raft_consensus.cc:348] T 523d9fb810d440d5ba1b16fd3da57602 P 16464c128b8649d6bafdf4f45f981031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.744428 15127 raft_consensus.cc:729] T 523d9fb810d440d5ba1b16fd3da57602 P 16464c128b8649d6bafdf4f45f981031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16464c128b8649d6bafdf4f45f981031, State: Initialized, Role: FOLLOWER
I20260924 12:48:55.744436 15129 raft_consensus.cc:729] T d396cddfe7674886b5ba7519d8495bf3 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af5894d0b4fe4f3ea1ebb39472de0eb7, State: Initialized, Role: FOLLOWER
I20260924 12:48:55.744515 15126 tablet_bootstrap.cc:492] T d011c630724442dfab74007165a31164 P 07ae11c0def5467ea650240db0071d78: No bootstrap required, opened a new log
I20260924 12:48:55.744679 15126 ts_tablet_manager.cc:1397] T d011c630724442dfab74007165a31164 P 07ae11c0def5467ea650240db0071d78: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.000s
I20260924 12:48:55.744691 15127 consensus_queue.cc:260] T 523d9fb810d440d5ba1b16fd3da57602 P 16464c128b8649d6bafdf4f45f981031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.744720 15129 consensus_queue.cc:260] T d396cddfe7674886b5ba7519d8495bf3 P af5894d0b4fe4f3ea1ebb39472de0eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.745409 15127 ts_tablet_manager.cc:1428] T 523d9fb810d440d5ba1b16fd3da57602 P 16464c128b8649d6bafdf4f45f981031: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 12:48:55.745419 15129 ts_tablet_manager.cc:1428] T d396cddfe7674886b5ba7519d8495bf3 P af5894d0b4fe4f3ea1ebb39472de0eb7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 12:48:55.745455 15126 raft_consensus.cc:348] T d011c630724442dfab74007165a31164 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.745728 15129 tablet_bootstrap.cc:492] T 75c3b8a29aa1489da08a21b6cf353bc6 P af5894d0b4fe4f3ea1ebb39472de0eb7: Bootstrap starting.
I20260924 12:48:55.745710 15126 raft_consensus.cc:729] T d011c630724442dfab74007165a31164 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae11c0def5467ea650240db0071d78, State: Initialized, Role: FOLLOWER
I20260924 12:48:55.745740 15127 tablet_bootstrap.cc:492] T 9723e825f07d48c9bc9fd927dd8ce48e P 16464c128b8649d6bafdf4f45f981031: Bootstrap starting.
I20260924 12:48:55.745888 15126 consensus_queue.cc:260] T d011c630724442dfab74007165a31164 P 07ae11c0def5467ea650240db0071d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.746404 15126 ts_tablet_manager.cc:1428] T d011c630724442dfab74007165a31164 P 07ae11c0def5467ea650240db0071d78: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 12:48:55.746722 15126 tablet_bootstrap.cc:492] T 3dd93acfeb704bc6b84977a1f6946aa3 P 07ae11c0def5467ea650240db0071d78: Bootstrap starting.
I20260924 12:48:55.747565 15127 tablet_bootstrap.cc:654] T 9723e825f07d48c9bc9fd927dd8ce48e P 16464c128b8649d6bafdf4f45f981031: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:55.748304 15126 tablet_bootstrap.cc:654] T 3dd93acfeb704bc6b84977a1f6946aa3 P 07ae11c0def5467ea650240db0071d78: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:55.748304 15129 tablet_bootstrap.cc:654] T 75c3b8a29aa1489da08a21b6cf353bc6 P af5894d0b4fe4f3ea1ebb39472de0eb7: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:55.753728 15126 tablet_bootstrap.cc:492] T 3dd93acfeb704bc6b84977a1f6946aa3 P 07ae11c0def5467ea650240db0071d78: No bootstrap required, opened a new log
I20260924 12:48:55.753913 15126 ts_tablet_manager.cc:1397] T 3dd93acfeb704bc6b84977a1f6946aa3 P 07ae11c0def5467ea650240db0071d78: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260924 12:48:55.754678 15126 raft_consensus.cc:348] T 3dd93acfeb704bc6b84977a1f6946aa3 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.754938 15126 raft_consensus.cc:729] T 3dd93acfeb704bc6b84977a1f6946aa3 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae11c0def5467ea650240db0071d78, State: Initialized, Role: FOLLOWER
I20260924 12:48:55.755118 15126 consensus_queue.cc:260] T 3dd93acfeb704bc6b84977a1f6946aa3 P 07ae11c0def5467ea650240db0071d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.755774 15126 ts_tablet_manager.cc:1428] T 3dd93acfeb704bc6b84977a1f6946aa3 P 07ae11c0def5467ea650240db0071d78: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 12:48:55.756078 15126 tablet_bootstrap.cc:492] T 523d9fb810d440d5ba1b16fd3da57602 P 07ae11c0def5467ea650240db0071d78: Bootstrap starting.
W20260924 12:48:55.757498 15104 tablet.cc:2369] T d011c630724442dfab74007165a31164 P af5894d0b4fe4f3ea1ebb39472de0eb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:48:55.759210 15126 tablet_bootstrap.cc:654] T 523d9fb810d440d5ba1b16fd3da57602 P 07ae11c0def5467ea650240db0071d78: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:55.761447 15127 tablet_bootstrap.cc:492] T 9723e825f07d48c9bc9fd927dd8ce48e P 16464c128b8649d6bafdf4f45f981031: No bootstrap required, opened a new log
I20260924 12:48:55.761649 15127 ts_tablet_manager.cc:1397] T 9723e825f07d48c9bc9fd927dd8ce48e P 16464c128b8649d6bafdf4f45f981031: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.015s
I20260924 12:48:55.762418 15127 raft_consensus.cc:348] T 9723e825f07d48c9bc9fd927dd8ce48e P 16464c128b8649d6bafdf4f45f981031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.762602 15129 tablet_bootstrap.cc:492] T 75c3b8a29aa1489da08a21b6cf353bc6 P af5894d0b4fe4f3ea1ebb39472de0eb7: No bootstrap required, opened a new log
I20260924 12:48:55.762737 15127 raft_consensus.cc:729] T 9723e825f07d48c9bc9fd927dd8ce48e P 16464c128b8649d6bafdf4f45f981031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16464c128b8649d6bafdf4f45f981031, State: Initialized, Role: FOLLOWER
I20260924 12:48:55.762821 15129 ts_tablet_manager.cc:1397] T 75c3b8a29aa1489da08a21b6cf353bc6 P af5894d0b4fe4f3ea1ebb39472de0eb7: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
I20260924 12:48:55.763003 15127 consensus_queue.cc:260] T 9723e825f07d48c9bc9fd927dd8ce48e P 16464c128b8649d6bafdf4f45f981031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.763691 15127 ts_tablet_manager.cc:1428] T 9723e825f07d48c9bc9fd927dd8ce48e P 16464c128b8649d6bafdf4f45f981031: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260924 12:48:55.763777 15126 tablet_bootstrap.cc:492] T 523d9fb810d440d5ba1b16fd3da57602 P 07ae11c0def5467ea650240db0071d78: No bootstrap required, opened a new log
I20260924 12:48:55.763651 15129 raft_consensus.cc:348] T 75c3b8a29aa1489da08a21b6cf353bc6 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.763934 15126 ts_tablet_manager.cc:1397] T 523d9fb810d440d5ba1b16fd3da57602 P 07ae11c0def5467ea650240db0071d78: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20260924 12:48:55.763972 15129 raft_consensus.cc:729] T 75c3b8a29aa1489da08a21b6cf353bc6 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af5894d0b4fe4f3ea1ebb39472de0eb7, State: Initialized, Role: FOLLOWER
I20260924 12:48:55.764153 15127 tablet_bootstrap.cc:492] T 75c3b8a29aa1489da08a21b6cf353bc6 P 16464c128b8649d6bafdf4f45f981031: Bootstrap starting.
I20260924 12:48:55.764235 15129 consensus_queue.cc:260] T 75c3b8a29aa1489da08a21b6cf353bc6 P af5894d0b4fe4f3ea1ebb39472de0eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.764652 15126 raft_consensus.cc:348] T 523d9fb810d440d5ba1b16fd3da57602 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.764889 15126 raft_consensus.cc:729] T 523d9fb810d440d5ba1b16fd3da57602 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae11c0def5467ea650240db0071d78, State: Initialized, Role: FOLLOWER
I20260924 12:48:55.764931 15129 ts_tablet_manager.cc:1428] T 75c3b8a29aa1489da08a21b6cf353bc6 P af5894d0b4fe4f3ea1ebb39472de0eb7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 12:48:55.765223 15129 tablet_bootstrap.cc:492] T 6e2027d642db4900bea24b47baf00757 P af5894d0b4fe4f3ea1ebb39472de0eb7: Bootstrap starting.
I20260924 12:48:55.765092 15126 consensus_queue.cc:260] T 523d9fb810d440d5ba1b16fd3da57602 P 07ae11c0def5467ea650240db0071d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.765944 15127 tablet_bootstrap.cc:654] T 75c3b8a29aa1489da08a21b6cf353bc6 P 16464c128b8649d6bafdf4f45f981031: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:55.766134 15126 ts_tablet_manager.cc:1428] T 523d9fb810d440d5ba1b16fd3da57602 P 07ae11c0def5467ea650240db0071d78: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 12:48:55.766396 15126 tablet_bootstrap.cc:492] T 2a500eb27ec84386ac2e26638a9c4221 P 07ae11c0def5467ea650240db0071d78: Bootstrap starting.
I20260924 12:48:55.767352 15129 tablet_bootstrap.cc:654] T 6e2027d642db4900bea24b47baf00757 P af5894d0b4fe4f3ea1ebb39472de0eb7: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:55.767972 15126 tablet_bootstrap.cc:654] T 2a500eb27ec84386ac2e26638a9c4221 P 07ae11c0def5467ea650240db0071d78: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:55.772509 15129 tablet_bootstrap.cc:492] T 6e2027d642db4900bea24b47baf00757 P af5894d0b4fe4f3ea1ebb39472de0eb7: No bootstrap required, opened a new log
I20260924 12:48:55.772737 15129 ts_tablet_manager.cc:1397] T 6e2027d642db4900bea24b47baf00757 P af5894d0b4fe4f3ea1ebb39472de0eb7: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260924 12:48:55.773586 15129 raft_consensus.cc:348] T 6e2027d642db4900bea24b47baf00757 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.773903 15129 raft_consensus.cc:729] T 6e2027d642db4900bea24b47baf00757 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af5894d0b4fe4f3ea1ebb39472de0eb7, State: Initialized, Role: FOLLOWER
I20260924 12:48:55.774121 15129 consensus_queue.cc:260] T 6e2027d642db4900bea24b47baf00757 P af5894d0b4fe4f3ea1ebb39472de0eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.774857 15129 ts_tablet_manager.cc:1428] T 6e2027d642db4900bea24b47baf00757 P af5894d0b4fe4f3ea1ebb39472de0eb7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260924 12:48:55.775204 15129 tablet_bootstrap.cc:492] T 5c66212111554f04893157d46ea41a71 P af5894d0b4fe4f3ea1ebb39472de0eb7: Bootstrap starting.
I20260924 12:48:55.776978 15129 tablet_bootstrap.cc:654] T 5c66212111554f04893157d46ea41a71 P af5894d0b4fe4f3ea1ebb39472de0eb7: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:55.778429 15127 tablet_bootstrap.cc:492] T 75c3b8a29aa1489da08a21b6cf353bc6 P 16464c128b8649d6bafdf4f45f981031: No bootstrap required, opened a new log
I20260924 12:48:55.778685 15127 ts_tablet_manager.cc:1397] T 75c3b8a29aa1489da08a21b6cf353bc6 P 16464c128b8649d6bafdf4f45f981031: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.011s
I20260924 12:48:55.779597 15127 raft_consensus.cc:348] T 75c3b8a29aa1489da08a21b6cf353bc6 P 16464c128b8649d6bafdf4f45f981031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.779806 15126 tablet_bootstrap.cc:492] T 2a500eb27ec84386ac2e26638a9c4221 P 07ae11c0def5467ea650240db0071d78: No bootstrap required, opened a new log
I20260924 12:48:55.779857 15127 raft_consensus.cc:729] T 75c3b8a29aa1489da08a21b6cf353bc6 P 16464c128b8649d6bafdf4f45f981031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16464c128b8649d6bafdf4f45f981031, State: Initialized, Role: FOLLOWER
I20260924 12:48:55.779978 15126 ts_tablet_manager.cc:1397] T 2a500eb27ec84386ac2e26638a9c4221 P 07ae11c0def5467ea650240db0071d78: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.003s
I20260924 12:48:55.780046 15127 consensus_queue.cc:260] T 75c3b8a29aa1489da08a21b6cf353bc6 P 16464c128b8649d6bafdf4f45f981031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.780507 15127 ts_tablet_manager.cc:1428] T 75c3b8a29aa1489da08a21b6cf353bc6 P 16464c128b8649d6bafdf4f45f981031: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 12:48:55.780755 15127 tablet_bootstrap.cc:492] T 3dd93acfeb704bc6b84977a1f6946aa3 P 16464c128b8649d6bafdf4f45f981031: Bootstrap starting.
I20260924 12:48:55.780709 15126 raft_consensus.cc:348] T 2a500eb27ec84386ac2e26638a9c4221 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.780941 15126 raft_consensus.cc:729] T 2a500eb27ec84386ac2e26638a9c4221 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae11c0def5467ea650240db0071d78, State: Initialized, Role: FOLLOWER
I20260924 12:48:55.781116 15126 consensus_queue.cc:260] T 2a500eb27ec84386ac2e26638a9c4221 P 07ae11c0def5467ea650240db0071d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.781620 15126 ts_tablet_manager.cc:1428] T 2a500eb27ec84386ac2e26638a9c4221 P 07ae11c0def5467ea650240db0071d78: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 12:48:55.781934 15126 tablet_bootstrap.cc:492] T d396cddfe7674886b5ba7519d8495bf3 P 07ae11c0def5467ea650240db0071d78: Bootstrap starting.
I20260924 12:48:55.782371 15127 tablet_bootstrap.cc:654] T 3dd93acfeb704bc6b84977a1f6946aa3 P 16464c128b8649d6bafdf4f45f981031: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:55.782727 15129 tablet_bootstrap.cc:492] T 5c66212111554f04893157d46ea41a71 P af5894d0b4fe4f3ea1ebb39472de0eb7: No bootstrap required, opened a new log
I20260924 12:48:55.782893 15129 ts_tablet_manager.cc:1397] T 5c66212111554f04893157d46ea41a71 P af5894d0b4fe4f3ea1ebb39472de0eb7: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260924 12:48:55.783593 15126 tablet_bootstrap.cc:654] T d396cddfe7674886b5ba7519d8495bf3 P 07ae11c0def5467ea650240db0071d78: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:55.783583 15129 raft_consensus.cc:348] T 5c66212111554f04893157d46ea41a71 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.783809 15129 raft_consensus.cc:729] T 5c66212111554f04893157d46ea41a71 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af5894d0b4fe4f3ea1ebb39472de0eb7, State: Initialized, Role: FOLLOWER
I20260924 12:48:55.783981 15129 consensus_queue.cc:260] T 5c66212111554f04893157d46ea41a71 P af5894d0b4fe4f3ea1ebb39472de0eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.785691 15129 ts_tablet_manager.cc:1428] T 5c66212111554f04893157d46ea41a71 P af5894d0b4fe4f3ea1ebb39472de0eb7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 12:48:55.785957 15129 tablet_bootstrap.cc:492] T 3dd93acfeb704bc6b84977a1f6946aa3 P af5894d0b4fe4f3ea1ebb39472de0eb7: Bootstrap starting.
I20260924 12:48:55.787657 15129 tablet_bootstrap.cc:654] T 3dd93acfeb704bc6b84977a1f6946aa3 P af5894d0b4fe4f3ea1ebb39472de0eb7: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:55.794237 15126 tablet_bootstrap.cc:492] T d396cddfe7674886b5ba7519d8495bf3 P 07ae11c0def5467ea650240db0071d78: No bootstrap required, opened a new log
I20260924 12:48:55.794430 15126 ts_tablet_manager.cc:1397] T d396cddfe7674886b5ba7519d8495bf3 P 07ae11c0def5467ea650240db0071d78: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260924 12:48:55.794636 15129 tablet_bootstrap.cc:492] T 3dd93acfeb704bc6b84977a1f6946aa3 P af5894d0b4fe4f3ea1ebb39472de0eb7: No bootstrap required, opened a new log
I20260924 12:48:55.794802 15129 ts_tablet_manager.cc:1397] T 3dd93acfeb704bc6b84977a1f6946aa3 P af5894d0b4fe4f3ea1ebb39472de0eb7: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 12:48:55.795169 15127 tablet_bootstrap.cc:492] T 3dd93acfeb704bc6b84977a1f6946aa3 P 16464c128b8649d6bafdf4f45f981031: No bootstrap required, opened a new log
I20260924 12:48:55.795341 15127 ts_tablet_manager.cc:1397] T 3dd93acfeb704bc6b84977a1f6946aa3 P 16464c128b8649d6bafdf4f45f981031: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.001s
I20260924 12:48:55.795239 15126 raft_consensus.cc:348] T d396cddfe7674886b5ba7519d8495bf3 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.795498 15126 raft_consensus.cc:729] T d396cddfe7674886b5ba7519d8495bf3 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae11c0def5467ea650240db0071d78, State: Initialized, Role: FOLLOWER
I20260924 12:48:55.795385 15129 raft_consensus.cc:348] T 3dd93acfeb704bc6b84977a1f6946aa3 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.795614 15129 raft_consensus.cc:729] T 3dd93acfeb704bc6b84977a1f6946aa3 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af5894d0b4fe4f3ea1ebb39472de0eb7, State: Initialized, Role: FOLLOWER
I20260924 12:48:55.795681 15126 consensus_queue.cc:260] T d396cddfe7674886b5ba7519d8495bf3 P 07ae11c0def5467ea650240db0071d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.795795 15129 consensus_queue.cc:260] T 3dd93acfeb704bc6b84977a1f6946aa3 P af5894d0b4fe4f3ea1ebb39472de0eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.796025 15127 raft_consensus.cc:348] T 3dd93acfeb704bc6b84977a1f6946aa3 P 16464c128b8649d6bafdf4f45f981031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.796257 15127 raft_consensus.cc:729] T 3dd93acfeb704bc6b84977a1f6946aa3 P 16464c128b8649d6bafdf4f45f981031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16464c128b8649d6bafdf4f45f981031, State: Initialized, Role: FOLLOWER
I20260924 12:48:55.796345 15126 ts_tablet_manager.cc:1428] T d396cddfe7674886b5ba7519d8495bf3 P 07ae11c0def5467ea650240db0071d78: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 12:48:55.796424 15127 consensus_queue.cc:260] T 3dd93acfeb704bc6b84977a1f6946aa3 P 16464c128b8649d6bafdf4f45f981031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.796943 15127 ts_tablet_manager.cc:1428] T 3dd93acfeb704bc6b84977a1f6946aa3 P 16464c128b8649d6bafdf4f45f981031: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 12:48:55.797204 15127 tablet_bootstrap.cc:492] T d396cddfe7674886b5ba7519d8495bf3 P 16464c128b8649d6bafdf4f45f981031: Bootstrap starting.
I20260924 12:48:55.796602 15126 tablet_bootstrap.cc:492] T e9f03002a94945c692b7b26f332d95e8 P 07ae11c0def5467ea650240db0071d78: Bootstrap starting.
I20260924 12:48:55.798870 15127 tablet_bootstrap.cc:654] T d396cddfe7674886b5ba7519d8495bf3 P 16464c128b8649d6bafdf4f45f981031: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:55.798866 15129 ts_tablet_manager.cc:1428] T 3dd93acfeb704bc6b84977a1f6946aa3 P af5894d0b4fe4f3ea1ebb39472de0eb7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260924 12:48:55.799440 15126 tablet_bootstrap.cc:654] T e9f03002a94945c692b7b26f332d95e8 P 07ae11c0def5467ea650240db0071d78: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:55.801239 15129 tablet_bootstrap.cc:492] T 9723e825f07d48c9bc9fd927dd8ce48e P af5894d0b4fe4f3ea1ebb39472de0eb7: Bootstrap starting.
I20260924 12:48:55.802855 15129 tablet_bootstrap.cc:654] T 9723e825f07d48c9bc9fd927dd8ce48e P af5894d0b4fe4f3ea1ebb39472de0eb7: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:55.807336 15126 tablet_bootstrap.cc:492] T e9f03002a94945c692b7b26f332d95e8 P 07ae11c0def5467ea650240db0071d78: No bootstrap required, opened a new log
I20260924 12:48:55.807335 15127 tablet_bootstrap.cc:492] T d396cddfe7674886b5ba7519d8495bf3 P 16464c128b8649d6bafdf4f45f981031: No bootstrap required, opened a new log
I20260924 12:48:55.807503 15127 ts_tablet_manager.cc:1397] T d396cddfe7674886b5ba7519d8495bf3 P 16464c128b8649d6bafdf4f45f981031: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20260924 12:48:55.807502 15126 ts_tablet_manager.cc:1397] T e9f03002a94945c692b7b26f332d95e8 P 07ae11c0def5467ea650240db0071d78: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260924 12:48:55.808202 15127 raft_consensus.cc:348] T d396cddfe7674886b5ba7519d8495bf3 P 16464c128b8649d6bafdf4f45f981031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.808202 15126 raft_consensus.cc:348] T e9f03002a94945c692b7b26f332d95e8 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } }
I20260924 12:48:55.808439 15127 raft_consensus.cc:729] T d396cddfe7674886b5ba7519d8495bf3 P 16464c128b8649d6bafdf4f45f981031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16464c128b8649d6bafdf4f45f981031, State: Initialized, Role: FOLLOWER
I20260924 12:48:55.808439 15126 raft_consensus.cc:729] T e9f03002a94945c692b7b26f332d95e8 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae11c0def5467ea650240db0071d78, State: Initialized, Role: FOLLOWER
I20260924 12:48:55.808619 15126 consensus_queue.cc:260] T e9f03002a94945c692b7b26f332d95e8 P 07ae11c0def5467ea650240db0071d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } }
I20260924 12:48:55.808625 15127 consensus_queue.cc:260] T d396cddfe7674886b5ba7519d8495bf3 P 16464c128b8649d6bafdf4f45f981031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.809195 15126 ts_tablet_manager.cc:1428] T e9f03002a94945c692b7b26f332d95e8 P 07ae11c0def5467ea650240db0071d78: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 12:48:55.809221 15127 ts_tablet_manager.cc:1428] T d396cddfe7674886b5ba7519d8495bf3 P 16464c128b8649d6bafdf4f45f981031: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260924 12:48:55.809525 15126 tablet_bootstrap.cc:492] T 5c66212111554f04893157d46ea41a71 P 07ae11c0def5467ea650240db0071d78: Bootstrap starting.
I20260924 12:48:55.809965 15127 tablet_bootstrap.cc:492] T e9f03002a94945c692b7b26f332d95e8 P 16464c128b8649d6bafdf4f45f981031: Bootstrap starting.
I20260924 12:48:55.811144 15126 tablet_bootstrap.cc:654] T 5c66212111554f04893157d46ea41a71 P 07ae11c0def5467ea650240db0071d78: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:55.811460 15127 tablet_bootstrap.cc:654] T e9f03002a94945c692b7b26f332d95e8 P 16464c128b8649d6bafdf4f45f981031: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:55.814422 15129 tablet_bootstrap.cc:492] T 9723e825f07d48c9bc9fd927dd8ce48e P af5894d0b4fe4f3ea1ebb39472de0eb7: No bootstrap required, opened a new log
I20260924 12:48:55.814620 15129 ts_tablet_manager.cc:1397] T 9723e825f07d48c9bc9fd927dd8ce48e P af5894d0b4fe4f3ea1ebb39472de0eb7: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.002s
I20260924 12:48:55.815389 15129 raft_consensus.cc:348] T 9723e825f07d48c9bc9fd927dd8ce48e P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.815614 15129 raft_consensus.cc:729] T 9723e825f07d48c9bc9fd927dd8ce48e P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af5894d0b4fe4f3ea1ebb39472de0eb7, State: Initialized, Role: FOLLOWER
I20260924 12:48:55.815816 15129 consensus_queue.cc:260] T 9723e825f07d48c9bc9fd927dd8ce48e P af5894d0b4fe4f3ea1ebb39472de0eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.816319 15129 ts_tablet_manager.cc:1428] T 9723e825f07d48c9bc9fd927dd8ce48e P af5894d0b4fe4f3ea1ebb39472de0eb7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 12:48:55.816469 15127 tablet_bootstrap.cc:492] T e9f03002a94945c692b7b26f332d95e8 P 16464c128b8649d6bafdf4f45f981031: No bootstrap required, opened a new log
I20260924 12:48:55.816602 15129 tablet_bootstrap.cc:492] T 2a500eb27ec84386ac2e26638a9c4221 P af5894d0b4fe4f3ea1ebb39472de0eb7: Bootstrap starting.
I20260924 12:48:55.816636 15127 ts_tablet_manager.cc:1397] T e9f03002a94945c692b7b26f332d95e8 P 16464c128b8649d6bafdf4f45f981031: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 12:48:55.817440 15127 raft_consensus.cc:348] T e9f03002a94945c692b7b26f332d95e8 P 16464c128b8649d6bafdf4f45f981031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } }
I20260924 12:48:55.817696 15127 raft_consensus.cc:729] T e9f03002a94945c692b7b26f332d95e8 P 16464c128b8649d6bafdf4f45f981031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16464c128b8649d6bafdf4f45f981031, State: Initialized, Role: FOLLOWER
I20260924 12:48:55.817890 15127 consensus_queue.cc:260] T e9f03002a94945c692b7b26f332d95e8 P 16464c128b8649d6bafdf4f45f981031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } }
I20260924 12:48:55.818184 15129 tablet_bootstrap.cc:654] T 2a500eb27ec84386ac2e26638a9c4221 P af5894d0b4fe4f3ea1ebb39472de0eb7: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:55.818543 15127 ts_tablet_manager.cc:1428] T e9f03002a94945c692b7b26f332d95e8 P 16464c128b8649d6bafdf4f45f981031: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 12:48:55.818830 15127 tablet_bootstrap.cc:492] T 2a500eb27ec84386ac2e26638a9c4221 P 16464c128b8649d6bafdf4f45f981031: Bootstrap starting.
I20260924 12:48:55.819693 15126 tablet_bootstrap.cc:492] T 5c66212111554f04893157d46ea41a71 P 07ae11c0def5467ea650240db0071d78: No bootstrap required, opened a new log
I20260924 12:48:55.819857 15126 ts_tablet_manager.cc:1397] T 5c66212111554f04893157d46ea41a71 P 07ae11c0def5467ea650240db0071d78: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260924 12:48:55.820410 15127 tablet_bootstrap.cc:654] T 2a500eb27ec84386ac2e26638a9c4221 P 16464c128b8649d6bafdf4f45f981031: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:55.820568 15126 raft_consensus.cc:348] T 5c66212111554f04893157d46ea41a71 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.820809 15126 raft_consensus.cc:729] T 5c66212111554f04893157d46ea41a71 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae11c0def5467ea650240db0071d78, State: Initialized, Role: FOLLOWER
I20260924 12:48:55.820994 15126 consensus_queue.cc:260] T 5c66212111554f04893157d46ea41a71 P 07ae11c0def5467ea650240db0071d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.822201 15126 ts_tablet_manager.cc:1428] T 5c66212111554f04893157d46ea41a71 P 07ae11c0def5467ea650240db0071d78: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260924 12:48:55.822909 14974 tablet.cc:2369] T 2a500eb27ec84386ac2e26638a9c4221 P 07ae11c0def5467ea650240db0071d78: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:48:55.824975 15127 tablet_bootstrap.cc:492] T 2a500eb27ec84386ac2e26638a9c4221 P 16464c128b8649d6bafdf4f45f981031: No bootstrap required, opened a new log
I20260924 12:48:55.825093 15127 ts_tablet_manager.cc:1397] T 2a500eb27ec84386ac2e26638a9c4221 P 16464c128b8649d6bafdf4f45f981031: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 12:48:55.825670 15127 raft_consensus.cc:348] T 2a500eb27ec84386ac2e26638a9c4221 P 16464c128b8649d6bafdf4f45f981031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.825892 15127 raft_consensus.cc:729] T 2a500eb27ec84386ac2e26638a9c4221 P 16464c128b8649d6bafdf4f45f981031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16464c128b8649d6bafdf4f45f981031, State: Initialized, Role: FOLLOWER
I20260924 12:48:55.826059 15127 consensus_queue.cc:260] T 2a500eb27ec84386ac2e26638a9c4221 P 16464c128b8649d6bafdf4f45f981031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.826651 15127 ts_tablet_manager.cc:1428] T 2a500eb27ec84386ac2e26638a9c4221 P 16464c128b8649d6bafdf4f45f981031: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 12:48:55.826871 15127 tablet_bootstrap.cc:492] T 5c66212111554f04893157d46ea41a71 P 16464c128b8649d6bafdf4f45f981031: Bootstrap starting.
I20260924 12:48:55.828227 15129 tablet_bootstrap.cc:492] T 2a500eb27ec84386ac2e26638a9c4221 P af5894d0b4fe4f3ea1ebb39472de0eb7: No bootstrap required, opened a new log
I20260924 12:48:55.828297 15127 tablet_bootstrap.cc:654] T 5c66212111554f04893157d46ea41a71 P 16464c128b8649d6bafdf4f45f981031: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:55.828404 15129 ts_tablet_manager.cc:1397] T 2a500eb27ec84386ac2e26638a9c4221 P af5894d0b4fe4f3ea1ebb39472de0eb7: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20260924 12:48:55.829116 15129 raft_consensus.cc:348] T 2a500eb27ec84386ac2e26638a9c4221 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.829377 15129 raft_consensus.cc:729] T 2a500eb27ec84386ac2e26638a9c4221 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af5894d0b4fe4f3ea1ebb39472de0eb7, State: Initialized, Role: FOLLOWER
I20260924 12:48:55.829550 15129 consensus_queue.cc:260] T 2a500eb27ec84386ac2e26638a9c4221 P af5894d0b4fe4f3ea1ebb39472de0eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.830049 15129 ts_tablet_manager.cc:1428] T 2a500eb27ec84386ac2e26638a9c4221 P af5894d0b4fe4f3ea1ebb39472de0eb7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 12:48:55.830225 15127 tablet_bootstrap.cc:492] T 5c66212111554f04893157d46ea41a71 P 16464c128b8649d6bafdf4f45f981031: No bootstrap required, opened a new log
I20260924 12:48:55.830333 15129 tablet_bootstrap.cc:492] T 523d9fb810d440d5ba1b16fd3da57602 P af5894d0b4fe4f3ea1ebb39472de0eb7: Bootstrap starting.
I20260924 12:48:55.830370 15127 ts_tablet_manager.cc:1397] T 5c66212111554f04893157d46ea41a71 P 16464c128b8649d6bafdf4f45f981031: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 12:48:55.831050 15127 raft_consensus.cc:348] T 5c66212111554f04893157d46ea41a71 P 16464c128b8649d6bafdf4f45f981031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.831295 15127 raft_consensus.cc:729] T 5c66212111554f04893157d46ea41a71 P 16464c128b8649d6bafdf4f45f981031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16464c128b8649d6bafdf4f45f981031, State: Initialized, Role: FOLLOWER
I20260924 12:48:55.831468 15127 consensus_queue.cc:260] T 5c66212111554f04893157d46ea41a71 P 16464c128b8649d6bafdf4f45f981031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.831983 15129 tablet_bootstrap.cc:654] T 523d9fb810d440d5ba1b16fd3da57602 P af5894d0b4fe4f3ea1ebb39472de0eb7: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:55.832089 15127 ts_tablet_manager.cc:1428] T 5c66212111554f04893157d46ea41a71 P 16464c128b8649d6bafdf4f45f981031: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 12:48:55.832378 15127 tablet_bootstrap.cc:492] T 6e2027d642db4900bea24b47baf00757 P 16464c128b8649d6bafdf4f45f981031: Bootstrap starting.
I20260924 12:48:55.834161 15127 tablet_bootstrap.cc:654] T 6e2027d642db4900bea24b47baf00757 P 16464c128b8649d6bafdf4f45f981031: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:55.835002 15129 tablet_bootstrap.cc:492] T 523d9fb810d440d5ba1b16fd3da57602 P af5894d0b4fe4f3ea1ebb39472de0eb7: No bootstrap required, opened a new log
I20260924 12:48:55.835161 15129 ts_tablet_manager.cc:1397] T 523d9fb810d440d5ba1b16fd3da57602 P af5894d0b4fe4f3ea1ebb39472de0eb7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 12:48:55.835808 15129 raft_consensus.cc:348] T 523d9fb810d440d5ba1b16fd3da57602 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.836041 15129 raft_consensus.cc:729] T 523d9fb810d440d5ba1b16fd3da57602 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af5894d0b4fe4f3ea1ebb39472de0eb7, State: Initialized, Role: FOLLOWER
I20260924 12:48:55.836225 15127 tablet_bootstrap.cc:492] T 6e2027d642db4900bea24b47baf00757 P 16464c128b8649d6bafdf4f45f981031: No bootstrap required, opened a new log
I20260924 12:48:55.836366 15127 ts_tablet_manager.cc:1397] T 6e2027d642db4900bea24b47baf00757 P 16464c128b8649d6bafdf4f45f981031: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260924 12:48:55.836228 15129 consensus_queue.cc:260] T 523d9fb810d440d5ba1b16fd3da57602 P af5894d0b4fe4f3ea1ebb39472de0eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.836787 15129 ts_tablet_manager.cc:1428] T 523d9fb810d440d5ba1b16fd3da57602 P af5894d0b4fe4f3ea1ebb39472de0eb7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 12:48:55.837090 15129 tablet_bootstrap.cc:492] T e9f03002a94945c692b7b26f332d95e8 P af5894d0b4fe4f3ea1ebb39472de0eb7: Bootstrap starting.
I20260924 12:48:55.837028 15127 raft_consensus.cc:348] T 6e2027d642db4900bea24b47baf00757 P 16464c128b8649d6bafdf4f45f981031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.837250 15127 raft_consensus.cc:729] T 6e2027d642db4900bea24b47baf00757 P 16464c128b8649d6bafdf4f45f981031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16464c128b8649d6bafdf4f45f981031, State: Initialized, Role: FOLLOWER
I20260924 12:48:55.837430 15127 consensus_queue.cc:260] T 6e2027d642db4900bea24b47baf00757 P 16464c128b8649d6bafdf4f45f981031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.838057 15127 ts_tablet_manager.cc:1428] T 6e2027d642db4900bea24b47baf00757 P 16464c128b8649d6bafdf4f45f981031: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260924 12:48:55.838690 15129 tablet_bootstrap.cc:654] T e9f03002a94945c692b7b26f332d95e8 P af5894d0b4fe4f3ea1ebb39472de0eb7: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:55.841034 15129 tablet_bootstrap.cc:492] T e9f03002a94945c692b7b26f332d95e8 P af5894d0b4fe4f3ea1ebb39472de0eb7: No bootstrap required, opened a new log
I20260924 12:48:55.841189 15129 ts_tablet_manager.cc:1397] T e9f03002a94945c692b7b26f332d95e8 P af5894d0b4fe4f3ea1ebb39472de0eb7: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260924 12:48:55.841799 15129 raft_consensus.cc:348] T e9f03002a94945c692b7b26f332d95e8 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } }
I20260924 12:48:55.842072 15129 raft_consensus.cc:729] T e9f03002a94945c692b7b26f332d95e8 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af5894d0b4fe4f3ea1ebb39472de0eb7, State: Initialized, Role: FOLLOWER
I20260924 12:48:55.842255 15129 consensus_queue.cc:260] T e9f03002a94945c692b7b26f332d95e8 P af5894d0b4fe4f3ea1ebb39472de0eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } }
I20260924 12:48:55.842692 15129 ts_tablet_manager.cc:1428] T e9f03002a94945c692b7b26f332d95e8 P af5894d0b4fe4f3ea1ebb39472de0eb7: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
W20260924 12:48:55.880888 14843 tablet.cc:2369] T 2a500eb27ec84386ac2e26638a9c4221 P 16464c128b8649d6bafdf4f45f981031: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:48:55.897455 14928 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6e2027d642db4900bea24b47baf00757"
dest_uuid: "07ae11c0def5467ea650240db0071d78"
from {username='slave'} at 127.0.0.1:52712
I20260924 12:48:55.897817 14928 raft_consensus.cc:482] T 6e2027d642db4900bea24b47baf00757 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 12:48:55.897967 14928 raft_consensus.cc:3037] T 6e2027d642db4900bea24b47baf00757 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:55.900061 14928 raft_consensus.cc:504] T 6e2027d642db4900bea24b47baf00757 P 07ae11c0def5467ea650240db0071d78 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.901472 14928 leader_election.cc:288] T 6e2027d642db4900bea24b47baf00757 P 07ae11c0def5467ea650240db0071d78 [CANDIDATE]: Term 1 election: Requested vote from peers 16464c128b8649d6bafdf4f45f981031 (127.5.77.193:37587), af5894d0b4fe4f3ea1ebb39472de0eb7 (127.5.77.195:38363)
I20260924 12:48:55.909248 15058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e2027d642db4900bea24b47baf00757" candidate_uuid: "07ae11c0def5467ea650240db0071d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7"
I20260924 12:48:55.909571 14797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e2027d642db4900bea24b47baf00757" candidate_uuid: "07ae11c0def5467ea650240db0071d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "16464c128b8649d6bafdf4f45f981031"
I20260924 12:48:55.909725 15058 raft_consensus.cc:3037] T 6e2027d642db4900bea24b47baf00757 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:55.909963 14797 raft_consensus.cc:3037] T 6e2027d642db4900bea24b47baf00757 P 16464c128b8649d6bafdf4f45f981031 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:55.912844 15058 raft_consensus.cc:2454] T 6e2027d642db4900bea24b47baf00757 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae11c0def5467ea650240db0071d78 in term 1.
I20260924 12:48:55.912844 14797 raft_consensus.cc:2454] T 6e2027d642db4900bea24b47baf00757 P 16464c128b8649d6bafdf4f45f981031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae11c0def5467ea650240db0071d78 in term 1.
I20260924 12:48:55.913527 14864 leader_election.cc:302] T 6e2027d642db4900bea24b47baf00757 P 07ae11c0def5467ea650240db0071d78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae11c0def5467ea650240db0071d78, 16464c128b8649d6bafdf4f45f981031; no voters:
I20260924 12:48:55.914001 15130 raft_consensus.cc:2781] T 6e2027d642db4900bea24b47baf00757 P 07ae11c0def5467ea650240db0071d78 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:55.914634 15130 raft_consensus.cc:686] T 6e2027d642db4900bea24b47baf00757 P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: Becoming Leader. State: Replica: 07ae11c0def5467ea650240db0071d78, State: Running, Role: LEADER
I20260924 12:48:55.915172 15130 consensus_queue.cc:237] T 6e2027d642db4900bea24b47baf00757 P 07ae11c0def5467ea650240db0071d78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.920243 14654 catalog_manager.cc:5495] T 6e2027d642db4900bea24b47baf00757 P 07ae11c0def5467ea650240db0071d78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07ae11c0def5467ea650240db0071d78 (127.5.77.194). New cstate: current_term: 1 leader_uuid: "07ae11c0def5467ea650240db0071d78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } health_report { overall_health: UNKNOWN } } }
I20260924 12:48:55.929311 14928 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2a500eb27ec84386ac2e26638a9c4221"
dest_uuid: "07ae11c0def5467ea650240db0071d78"
from {username='slave'} at 127.0.0.1:52712
I20260924 12:48:55.929567 14928 raft_consensus.cc:482] T 2a500eb27ec84386ac2e26638a9c4221 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 12:48:55.929670 14928 raft_consensus.cc:3037] T 2a500eb27ec84386ac2e26638a9c4221 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:55.931646 14928 raft_consensus.cc:504] T 2a500eb27ec84386ac2e26638a9c4221 P 07ae11c0def5467ea650240db0071d78 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.932197 14928 leader_election.cc:288] T 2a500eb27ec84386ac2e26638a9c4221 P 07ae11c0def5467ea650240db0071d78 [CANDIDATE]: Term 1 election: Requested vote from peers 16464c128b8649d6bafdf4f45f981031 (127.5.77.193:37587), af5894d0b4fe4f3ea1ebb39472de0eb7 (127.5.77.195:38363)
I20260924 12:48:55.932770 14797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a500eb27ec84386ac2e26638a9c4221" candidate_uuid: "07ae11c0def5467ea650240db0071d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "16464c128b8649d6bafdf4f45f981031"
I20260924 12:48:55.932786 15058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a500eb27ec84386ac2e26638a9c4221" candidate_uuid: "07ae11c0def5467ea650240db0071d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7"
I20260924 12:48:55.933008 15058 raft_consensus.cc:3037] T 2a500eb27ec84386ac2e26638a9c4221 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:55.933014 14797 raft_consensus.cc:3037] T 2a500eb27ec84386ac2e26638a9c4221 P 16464c128b8649d6bafdf4f45f981031 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:55.935151 15058 raft_consensus.cc:2454] T 2a500eb27ec84386ac2e26638a9c4221 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae11c0def5467ea650240db0071d78 in term 1.
I20260924 12:48:55.935201 14797 raft_consensus.cc:2454] T 2a500eb27ec84386ac2e26638a9c4221 P 16464c128b8649d6bafdf4f45f981031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae11c0def5467ea650240db0071d78 in term 1.
I20260924 12:48:55.935628 14862 leader_election.cc:302] T 2a500eb27ec84386ac2e26638a9c4221 P 07ae11c0def5467ea650240db0071d78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae11c0def5467ea650240db0071d78, af5894d0b4fe4f3ea1ebb39472de0eb7; no voters:
I20260924 12:48:55.935961 15130 raft_consensus.cc:2781] T 2a500eb27ec84386ac2e26638a9c4221 P 07ae11c0def5467ea650240db0071d78 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:55.936108 15130 raft_consensus.cc:686] T 2a500eb27ec84386ac2e26638a9c4221 P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: Becoming Leader. State: Replica: 07ae11c0def5467ea650240db0071d78, State: Running, Role: LEADER
I20260924 12:48:55.936360 15130 consensus_queue.cc:237] T 2a500eb27ec84386ac2e26638a9c4221 P 07ae11c0def5467ea650240db0071d78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.939034 14654 catalog_manager.cc:5495] T 2a500eb27ec84386ac2e26638a9c4221 P 07ae11c0def5467ea650240db0071d78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07ae11c0def5467ea650240db0071d78 (127.5.77.194). New cstate: current_term: 1 leader_uuid: "07ae11c0def5467ea650240db0071d78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } health_report { overall_health: UNKNOWN } } }
I20260924 12:48:55.948853 14928 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e9f03002a94945c692b7b26f332d95e8"
dest_uuid: "07ae11c0def5467ea650240db0071d78"
from {username='slave'} at 127.0.0.1:52712
I20260924 12:48:55.949059 14928 raft_consensus.cc:482] T e9f03002a94945c692b7b26f332d95e8 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 12:48:55.949186 14928 raft_consensus.cc:3037] T e9f03002a94945c692b7b26f332d95e8 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:55.951143 14928 raft_consensus.cc:504] T e9f03002a94945c692b7b26f332d95e8 P 07ae11c0def5467ea650240db0071d78 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } }
I20260924 12:48:55.951746 14928 leader_election.cc:288] T e9f03002a94945c692b7b26f332d95e8 P 07ae11c0def5467ea650240db0071d78 [CANDIDATE]: Term 1 election: Requested vote from peers 16464c128b8649d6bafdf4f45f981031 (127.5.77.193:37587), af5894d0b4fe4f3ea1ebb39472de0eb7 (127.5.77.195:38363)
I20260924 12:48:55.952231 14797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f03002a94945c692b7b26f332d95e8" candidate_uuid: "07ae11c0def5467ea650240db0071d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "16464c128b8649d6bafdf4f45f981031"
I20260924 12:48:55.952303 15058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f03002a94945c692b7b26f332d95e8" candidate_uuid: "07ae11c0def5467ea650240db0071d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7"
I20260924 12:48:55.952421 14797 raft_consensus.cc:3037] T e9f03002a94945c692b7b26f332d95e8 P 16464c128b8649d6bafdf4f45f981031 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:55.952489 15058 raft_consensus.cc:3037] T e9f03002a94945c692b7b26f332d95e8 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:55.954540 15058 raft_consensus.cc:2454] T e9f03002a94945c692b7b26f332d95e8 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae11c0def5467ea650240db0071d78 in term 1.
I20260924 12:48:55.954576 14797 raft_consensus.cc:2454] T e9f03002a94945c692b7b26f332d95e8 P 16464c128b8649d6bafdf4f45f981031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae11c0def5467ea650240db0071d78 in term 1.
I20260924 12:48:55.954988 14864 leader_election.cc:302] T e9f03002a94945c692b7b26f332d95e8 P 07ae11c0def5467ea650240db0071d78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 07ae11c0def5467ea650240db0071d78, 16464c128b8649d6bafdf4f45f981031, af5894d0b4fe4f3ea1ebb39472de0eb7; no voters:
I20260924 12:48:55.955272 15130 raft_consensus.cc:2781] T e9f03002a94945c692b7b26f332d95e8 P 07ae11c0def5467ea650240db0071d78 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:55.955410 15130 raft_consensus.cc:686] T e9f03002a94945c692b7b26f332d95e8 P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: Becoming Leader. State: Replica: 07ae11c0def5467ea650240db0071d78, State: Running, Role: LEADER
I20260924 12:48:55.955659 15130 consensus_queue.cc:237] T e9f03002a94945c692b7b26f332d95e8 P 07ae11c0def5467ea650240db0071d78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } }
I20260924 12:48:55.958401 14654 catalog_manager.cc:5495] T e9f03002a94945c692b7b26f332d95e8 P 07ae11c0def5467ea650240db0071d78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07ae11c0def5467ea650240db0071d78 (127.5.77.194). New cstate: current_term: 1 leader_uuid: "07ae11c0def5467ea650240db0071d78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } health_report { overall_health: HEALTHY } } }
I20260924 12:48:55.966440 14928 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "523d9fb810d440d5ba1b16fd3da57602"
dest_uuid: "07ae11c0def5467ea650240db0071d78"
from {username='slave'} at 127.0.0.1:52712
I20260924 12:48:55.966694 14928 raft_consensus.cc:482] T 523d9fb810d440d5ba1b16fd3da57602 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 12:48:55.966804 14928 raft_consensus.cc:3037] T 523d9fb810d440d5ba1b16fd3da57602 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:55.968858 14928 raft_consensus.cc:504] T 523d9fb810d440d5ba1b16fd3da57602 P 07ae11c0def5467ea650240db0071d78 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.969463 14928 leader_election.cc:288] T 523d9fb810d440d5ba1b16fd3da57602 P 07ae11c0def5467ea650240db0071d78 [CANDIDATE]: Term 1 election: Requested vote from peers 16464c128b8649d6bafdf4f45f981031 (127.5.77.193:37587), af5894d0b4fe4f3ea1ebb39472de0eb7 (127.5.77.195:38363)
I20260924 12:48:55.969980 14797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "523d9fb810d440d5ba1b16fd3da57602" candidate_uuid: "07ae11c0def5467ea650240db0071d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "16464c128b8649d6bafdf4f45f981031"
I20260924 12:48:55.970033 15058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "523d9fb810d440d5ba1b16fd3da57602" candidate_uuid: "07ae11c0def5467ea650240db0071d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7"
I20260924 12:48:55.970221 14797 raft_consensus.cc:3037] T 523d9fb810d440d5ba1b16fd3da57602 P 16464c128b8649d6bafdf4f45f981031 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:55.970261 15058 raft_consensus.cc:3037] T 523d9fb810d440d5ba1b16fd3da57602 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:55.972313 14797 raft_consensus.cc:2454] T 523d9fb810d440d5ba1b16fd3da57602 P 16464c128b8649d6bafdf4f45f981031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae11c0def5467ea650240db0071d78 in term 1.
I20260924 12:48:55.972354 15058 raft_consensus.cc:2454] T 523d9fb810d440d5ba1b16fd3da57602 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae11c0def5467ea650240db0071d78 in term 1.
I20260924 12:48:55.972743 14864 leader_election.cc:302] T 523d9fb810d440d5ba1b16fd3da57602 P 07ae11c0def5467ea650240db0071d78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae11c0def5467ea650240db0071d78, 16464c128b8649d6bafdf4f45f981031; no voters:
I20260924 12:48:55.972999 15130 raft_consensus.cc:2781] T 523d9fb810d440d5ba1b16fd3da57602 P 07ae11c0def5467ea650240db0071d78 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:55.973188 15130 raft_consensus.cc:686] T 523d9fb810d440d5ba1b16fd3da57602 P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: Becoming Leader. State: Replica: 07ae11c0def5467ea650240db0071d78, State: Running, Role: LEADER
I20260924 12:48:55.973419 15130 consensus_queue.cc:237] T 523d9fb810d440d5ba1b16fd3da57602 P 07ae11c0def5467ea650240db0071d78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.976045 14654 catalog_manager.cc:5495] T 523d9fb810d440d5ba1b16fd3da57602 P 07ae11c0def5467ea650240db0071d78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07ae11c0def5467ea650240db0071d78 (127.5.77.194). New cstate: current_term: 1 leader_uuid: "07ae11c0def5467ea650240db0071d78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } health_report { overall_health: UNKNOWN } } }
I20260924 12:48:55.984651 14928 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5c66212111554f04893157d46ea41a71"
dest_uuid: "07ae11c0def5467ea650240db0071d78"
from {username='slave'} at 127.0.0.1:52712
I20260924 12:48:55.984871 14928 raft_consensus.cc:482] T 5c66212111554f04893157d46ea41a71 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 12:48:55.984992 14928 raft_consensus.cc:3037] T 5c66212111554f04893157d46ea41a71 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:55.986896 14928 raft_consensus.cc:504] T 5c66212111554f04893157d46ea41a71 P 07ae11c0def5467ea650240db0071d78 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.987479 14928 leader_election.cc:288] T 5c66212111554f04893157d46ea41a71 P 07ae11c0def5467ea650240db0071d78 [CANDIDATE]: Term 1 election: Requested vote from peers 16464c128b8649d6bafdf4f45f981031 (127.5.77.193:37587), af5894d0b4fe4f3ea1ebb39472de0eb7 (127.5.77.195:38363)
I20260924 12:48:55.987920 14797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c66212111554f04893157d46ea41a71" candidate_uuid: "07ae11c0def5467ea650240db0071d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "16464c128b8649d6bafdf4f45f981031"
I20260924 12:48:55.988117 14797 raft_consensus.cc:3037] T 5c66212111554f04893157d46ea41a71 P 16464c128b8649d6bafdf4f45f981031 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:55.988054 15058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c66212111554f04893157d46ea41a71" candidate_uuid: "07ae11c0def5467ea650240db0071d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7"
I20260924 12:48:55.988252 15058 raft_consensus.cc:3037] T 5c66212111554f04893157d46ea41a71 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:55.990145 14797 raft_consensus.cc:2454] T 5c66212111554f04893157d46ea41a71 P 16464c128b8649d6bafdf4f45f981031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae11c0def5467ea650240db0071d78 in term 1.
I20260924 12:48:55.990186 15058 raft_consensus.cc:2454] T 5c66212111554f04893157d46ea41a71 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae11c0def5467ea650240db0071d78 in term 1.
I20260924 12:48:55.990589 14862 leader_election.cc:302] T 5c66212111554f04893157d46ea41a71 P 07ae11c0def5467ea650240db0071d78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae11c0def5467ea650240db0071d78, af5894d0b4fe4f3ea1ebb39472de0eb7; no voters:
I20260924 12:48:55.990892 15130 raft_consensus.cc:2781] T 5c66212111554f04893157d46ea41a71 P 07ae11c0def5467ea650240db0071d78 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:55.991034 15130 raft_consensus.cc:686] T 5c66212111554f04893157d46ea41a71 P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: Becoming Leader. State: Replica: 07ae11c0def5467ea650240db0071d78, State: Running, Role: LEADER
I20260924 12:48:55.991268 15130 consensus_queue.cc:237] T 5c66212111554f04893157d46ea41a71 P 07ae11c0def5467ea650240db0071d78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:55.993873 14654 catalog_manager.cc:5495] T 5c66212111554f04893157d46ea41a71 P 07ae11c0def5467ea650240db0071d78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07ae11c0def5467ea650240db0071d78 (127.5.77.194). New cstate: current_term: 1 leader_uuid: "07ae11c0def5467ea650240db0071d78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } health_report { overall_health: UNKNOWN } } }
I20260924 12:48:56.004243 14928 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d011c630724442dfab74007165a31164"
dest_uuid: "07ae11c0def5467ea650240db0071d78"
from {username='slave'} at 127.0.0.1:52712
I20260924 12:48:56.004523 14928 raft_consensus.cc:482] T d011c630724442dfab74007165a31164 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 12:48:56.004629 14928 raft_consensus.cc:3037] T d011c630724442dfab74007165a31164 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:56.006552 14928 raft_consensus.cc:504] T d011c630724442dfab74007165a31164 P 07ae11c0def5467ea650240db0071d78 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:56.007102 14928 leader_election.cc:288] T d011c630724442dfab74007165a31164 P 07ae11c0def5467ea650240db0071d78 [CANDIDATE]: Term 1 election: Requested vote from peers 16464c128b8649d6bafdf4f45f981031 (127.5.77.193:37587), af5894d0b4fe4f3ea1ebb39472de0eb7 (127.5.77.195:38363)
I20260924 12:48:56.007709 14797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d011c630724442dfab74007165a31164" candidate_uuid: "07ae11c0def5467ea650240db0071d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "16464c128b8649d6bafdf4f45f981031"
I20260924 12:48:56.007737 15058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d011c630724442dfab74007165a31164" candidate_uuid: "07ae11c0def5467ea650240db0071d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7"
I20260924 12:48:56.007936 14797 raft_consensus.cc:3037] T d011c630724442dfab74007165a31164 P 16464c128b8649d6bafdf4f45f981031 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:56.007999 15058 raft_consensus.cc:3037] T d011c630724442dfab74007165a31164 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:56.010120 14797 raft_consensus.cc:2454] T d011c630724442dfab74007165a31164 P 16464c128b8649d6bafdf4f45f981031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae11c0def5467ea650240db0071d78 in term 1.
I20260924 12:48:56.010121 15058 raft_consensus.cc:2454] T d011c630724442dfab74007165a31164 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae11c0def5467ea650240db0071d78 in term 1.
I20260924 12:48:56.010572 14862 leader_election.cc:302] T d011c630724442dfab74007165a31164 P 07ae11c0def5467ea650240db0071d78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae11c0def5467ea650240db0071d78, af5894d0b4fe4f3ea1ebb39472de0eb7; no voters:
I20260924 12:48:56.010828 15130 raft_consensus.cc:2781] T d011c630724442dfab74007165a31164 P 07ae11c0def5467ea650240db0071d78 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:56.010982 15130 raft_consensus.cc:686] T d011c630724442dfab74007165a31164 P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: Becoming Leader. State: Replica: 07ae11c0def5467ea650240db0071d78, State: Running, Role: LEADER
I20260924 12:48:56.011214 15130 consensus_queue.cc:237] T d011c630724442dfab74007165a31164 P 07ae11c0def5467ea650240db0071d78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:56.013779 14654 catalog_manager.cc:5495] T d011c630724442dfab74007165a31164 P 07ae11c0def5467ea650240db0071d78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07ae11c0def5467ea650240db0071d78 (127.5.77.194). New cstate: current_term: 1 leader_uuid: "07ae11c0def5467ea650240db0071d78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } health_report { overall_health: UNKNOWN } } }
I20260924 12:48:56.023522 14928 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3dd93acfeb704bc6b84977a1f6946aa3"
dest_uuid: "07ae11c0def5467ea650240db0071d78"
from {username='slave'} at 127.0.0.1:52712
I20260924 12:48:56.023773 14928 raft_consensus.cc:482] T 3dd93acfeb704bc6b84977a1f6946aa3 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 12:48:56.023890 14928 raft_consensus.cc:3037] T 3dd93acfeb704bc6b84977a1f6946aa3 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:56.025732 14928 raft_consensus.cc:504] T 3dd93acfeb704bc6b84977a1f6946aa3 P 07ae11c0def5467ea650240db0071d78 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:56.026335 14928 leader_election.cc:288] T 3dd93acfeb704bc6b84977a1f6946aa3 P 07ae11c0def5467ea650240db0071d78 [CANDIDATE]: Term 1 election: Requested vote from peers 16464c128b8649d6bafdf4f45f981031 (127.5.77.193:37587), af5894d0b4fe4f3ea1ebb39472de0eb7 (127.5.77.195:38363)
I20260924 12:48:56.026875 15058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dd93acfeb704bc6b84977a1f6946aa3" candidate_uuid: "07ae11c0def5467ea650240db0071d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7"
I20260924 12:48:56.026928 14797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dd93acfeb704bc6b84977a1f6946aa3" candidate_uuid: "07ae11c0def5467ea650240db0071d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "16464c128b8649d6bafdf4f45f981031"
I20260924 12:48:56.027110 15058 raft_consensus.cc:3037] T 3dd93acfeb704bc6b84977a1f6946aa3 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:56.027213 14797 raft_consensus.cc:3037] T 3dd93acfeb704bc6b84977a1f6946aa3 P 16464c128b8649d6bafdf4f45f981031 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:56.029106 15058 raft_consensus.cc:2454] T 3dd93acfeb704bc6b84977a1f6946aa3 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae11c0def5467ea650240db0071d78 in term 1.
I20260924 12:48:56.029209 14797 raft_consensus.cc:2454] T 3dd93acfeb704bc6b84977a1f6946aa3 P 16464c128b8649d6bafdf4f45f981031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae11c0def5467ea650240db0071d78 in term 1.
I20260924 12:48:56.029548 14862 leader_election.cc:302] T 3dd93acfeb704bc6b84977a1f6946aa3 P 07ae11c0def5467ea650240db0071d78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae11c0def5467ea650240db0071d78, af5894d0b4fe4f3ea1ebb39472de0eb7; no voters:
I20260924 12:48:56.029836 15130 raft_consensus.cc:2781] T 3dd93acfeb704bc6b84977a1f6946aa3 P 07ae11c0def5467ea650240db0071d78 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:56.029987 15130 raft_consensus.cc:686] T 3dd93acfeb704bc6b84977a1f6946aa3 P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: Becoming Leader. State: Replica: 07ae11c0def5467ea650240db0071d78, State: Running, Role: LEADER
I20260924 12:48:56.030236 15130 consensus_queue.cc:237] T 3dd93acfeb704bc6b84977a1f6946aa3 P 07ae11c0def5467ea650240db0071d78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:56.032809 14654 catalog_manager.cc:5495] T 3dd93acfeb704bc6b84977a1f6946aa3 P 07ae11c0def5467ea650240db0071d78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07ae11c0def5467ea650240db0071d78 (127.5.77.194). New cstate: current_term: 1 leader_uuid: "07ae11c0def5467ea650240db0071d78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } health_report { overall_health: UNKNOWN } } }
I20260924 12:48:56.041893 14928 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9723e825f07d48c9bc9fd927dd8ce48e"
dest_uuid: "07ae11c0def5467ea650240db0071d78"
from {username='slave'} at 127.0.0.1:52712
I20260924 12:48:56.042165 14928 raft_consensus.cc:482] T 9723e825f07d48c9bc9fd927dd8ce48e P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 12:48:56.042280 14928 raft_consensus.cc:3037] T 9723e825f07d48c9bc9fd927dd8ce48e P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:56.044399 14928 raft_consensus.cc:504] T 9723e825f07d48c9bc9fd927dd8ce48e P 07ae11c0def5467ea650240db0071d78 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:56.045083 14928 leader_election.cc:288] T 9723e825f07d48c9bc9fd927dd8ce48e P 07ae11c0def5467ea650240db0071d78 [CANDIDATE]: Term 1 election: Requested vote from peers 16464c128b8649d6bafdf4f45f981031 (127.5.77.193:37587), af5894d0b4fe4f3ea1ebb39472de0eb7 (127.5.77.195:38363)
I20260924 12:48:56.045624 15058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9723e825f07d48c9bc9fd927dd8ce48e" candidate_uuid: "07ae11c0def5467ea650240db0071d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7"
I20260924 12:48:56.045652 14797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9723e825f07d48c9bc9fd927dd8ce48e" candidate_uuid: "07ae11c0def5467ea650240db0071d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "16464c128b8649d6bafdf4f45f981031"
I20260924 12:48:56.045871 15058 raft_consensus.cc:3037] T 9723e825f07d48c9bc9fd927dd8ce48e P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:56.045931 14797 raft_consensus.cc:3037] T 9723e825f07d48c9bc9fd927dd8ce48e P 16464c128b8649d6bafdf4f45f981031 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:56.048100 15058 raft_consensus.cc:2454] T 9723e825f07d48c9bc9fd927dd8ce48e P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae11c0def5467ea650240db0071d78 in term 1.
I20260924 12:48:56.048285 14797 raft_consensus.cc:2454] T 9723e825f07d48c9bc9fd927dd8ce48e P 16464c128b8649d6bafdf4f45f981031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae11c0def5467ea650240db0071d78 in term 1.
I20260924 12:48:56.048533 14862 leader_election.cc:302] T 9723e825f07d48c9bc9fd927dd8ce48e P 07ae11c0def5467ea650240db0071d78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae11c0def5467ea650240db0071d78, af5894d0b4fe4f3ea1ebb39472de0eb7; no voters:
I20260924 12:48:56.048830 15130 raft_consensus.cc:2781] T 9723e825f07d48c9bc9fd927dd8ce48e P 07ae11c0def5467ea650240db0071d78 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:56.049016 15130 raft_consensus.cc:686] T 9723e825f07d48c9bc9fd927dd8ce48e P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: Becoming Leader. State: Replica: 07ae11c0def5467ea650240db0071d78, State: Running, Role: LEADER
I20260924 12:48:56.049268 15130 consensus_queue.cc:237] T 9723e825f07d48c9bc9fd927dd8ce48e P 07ae11c0def5467ea650240db0071d78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:56.051949 14654 catalog_manager.cc:5495] T 9723e825f07d48c9bc9fd927dd8ce48e P 07ae11c0def5467ea650240db0071d78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07ae11c0def5467ea650240db0071d78 (127.5.77.194). New cstate: current_term: 1 leader_uuid: "07ae11c0def5467ea650240db0071d78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } health_report { overall_health: UNKNOWN } } }
I20260924 12:48:56.061959 14928 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "75c3b8a29aa1489da08a21b6cf353bc6"
dest_uuid: "07ae11c0def5467ea650240db0071d78"
from {username='slave'} at 127.0.0.1:52712
I20260924 12:48:56.062165 14928 raft_consensus.cc:482] T 75c3b8a29aa1489da08a21b6cf353bc6 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 12:48:56.062265 14928 raft_consensus.cc:3037] T 75c3b8a29aa1489da08a21b6cf353bc6 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:56.064096 14928 raft_consensus.cc:504] T 75c3b8a29aa1489da08a21b6cf353bc6 P 07ae11c0def5467ea650240db0071d78 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:56.064651 14928 leader_election.cc:288] T 75c3b8a29aa1489da08a21b6cf353bc6 P 07ae11c0def5467ea650240db0071d78 [CANDIDATE]: Term 1 election: Requested vote from peers 16464c128b8649d6bafdf4f45f981031 (127.5.77.193:37587), af5894d0b4fe4f3ea1ebb39472de0eb7 (127.5.77.195:38363)
I20260924 12:48:56.065205 14797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75c3b8a29aa1489da08a21b6cf353bc6" candidate_uuid: "07ae11c0def5467ea650240db0071d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "16464c128b8649d6bafdf4f45f981031"
I20260924 12:48:56.065279 15058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75c3b8a29aa1489da08a21b6cf353bc6" candidate_uuid: "07ae11c0def5467ea650240db0071d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7"
I20260924 12:48:56.065429 14797 raft_consensus.cc:3037] T 75c3b8a29aa1489da08a21b6cf353bc6 P 16464c128b8649d6bafdf4f45f981031 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:56.065521 15058 raft_consensus.cc:3037] T 75c3b8a29aa1489da08a21b6cf353bc6 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:56.067801 14797 raft_consensus.cc:2454] T 75c3b8a29aa1489da08a21b6cf353bc6 P 16464c128b8649d6bafdf4f45f981031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae11c0def5467ea650240db0071d78 in term 1.
I20260924 12:48:56.068005 15058 raft_consensus.cc:2454] T 75c3b8a29aa1489da08a21b6cf353bc6 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae11c0def5467ea650240db0071d78 in term 1.
I20260924 12:48:56.068214 14864 leader_election.cc:302] T 75c3b8a29aa1489da08a21b6cf353bc6 P 07ae11c0def5467ea650240db0071d78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae11c0def5467ea650240db0071d78, 16464c128b8649d6bafdf4f45f981031; no voters:
I20260924 12:48:56.068565 15130 raft_consensus.cc:2781] T 75c3b8a29aa1489da08a21b6cf353bc6 P 07ae11c0def5467ea650240db0071d78 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:56.068732 15130 raft_consensus.cc:686] T 75c3b8a29aa1489da08a21b6cf353bc6 P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: Becoming Leader. State: Replica: 07ae11c0def5467ea650240db0071d78, State: Running, Role: LEADER
I20260924 12:48:56.069041 15130 consensus_queue.cc:237] T 75c3b8a29aa1489da08a21b6cf353bc6 P 07ae11c0def5467ea650240db0071d78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:56.071991 14654 catalog_manager.cc:5495] T 75c3b8a29aa1489da08a21b6cf353bc6 P 07ae11c0def5467ea650240db0071d78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07ae11c0def5467ea650240db0071d78 (127.5.77.194). New cstate: current_term: 1 leader_uuid: "07ae11c0def5467ea650240db0071d78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } health_report { overall_health: UNKNOWN } } }
I20260924 12:48:56.082355 14928 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d396cddfe7674886b5ba7519d8495bf3"
dest_uuid: "07ae11c0def5467ea650240db0071d78"
from {username='slave'} at 127.0.0.1:52712
I20260924 12:48:56.082641 14928 raft_consensus.cc:482] T d396cddfe7674886b5ba7519d8495bf3 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 12:48:56.082780 14928 raft_consensus.cc:3037] T d396cddfe7674886b5ba7519d8495bf3 P 07ae11c0def5467ea650240db0071d78 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:56.084833 14928 raft_consensus.cc:504] T d396cddfe7674886b5ba7519d8495bf3 P 07ae11c0def5467ea650240db0071d78 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:56.085433 14928 leader_election.cc:288] T d396cddfe7674886b5ba7519d8495bf3 P 07ae11c0def5467ea650240db0071d78 [CANDIDATE]: Term 1 election: Requested vote from peers 16464c128b8649d6bafdf4f45f981031 (127.5.77.193:37587), af5894d0b4fe4f3ea1ebb39472de0eb7 (127.5.77.195:38363)
I20260924 12:48:56.086009 15058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d396cddfe7674886b5ba7519d8495bf3" candidate_uuid: "07ae11c0def5467ea650240db0071d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7"
I20260924 12:48:56.086066 14797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d396cddfe7674886b5ba7519d8495bf3" candidate_uuid: "07ae11c0def5467ea650240db0071d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "16464c128b8649d6bafdf4f45f981031"
I20260924 12:48:56.086280 15058 raft_consensus.cc:3037] T d396cddfe7674886b5ba7519d8495bf3 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:56.086308 14797 raft_consensus.cc:3037] T d396cddfe7674886b5ba7519d8495bf3 P 16464c128b8649d6bafdf4f45f981031 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:56.089159 15058 raft_consensus.cc:2454] T d396cddfe7674886b5ba7519d8495bf3 P af5894d0b4fe4f3ea1ebb39472de0eb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae11c0def5467ea650240db0071d78 in term 1.
I20260924 12:48:56.089159 14797 raft_consensus.cc:2454] T d396cddfe7674886b5ba7519d8495bf3 P 16464c128b8649d6bafdf4f45f981031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae11c0def5467ea650240db0071d78 in term 1.
I20260924 12:48:56.089865 14862 leader_election.cc:302] T d396cddfe7674886b5ba7519d8495bf3 P 07ae11c0def5467ea650240db0071d78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 07ae11c0def5467ea650240db0071d78, 16464c128b8649d6bafdf4f45f981031, af5894d0b4fe4f3ea1ebb39472de0eb7; no voters:
I20260924 12:48:56.090247 15130 raft_consensus.cc:2781] T d396cddfe7674886b5ba7519d8495bf3 P 07ae11c0def5467ea650240db0071d78 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:56.090492 15130 raft_consensus.cc:686] T d396cddfe7674886b5ba7519d8495bf3 P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: Becoming Leader. State: Replica: 07ae11c0def5467ea650240db0071d78, State: Running, Role: LEADER
I20260924 12:48:56.090787 15130 consensus_queue.cc:237] T d396cddfe7674886b5ba7519d8495bf3 P 07ae11c0def5467ea650240db0071d78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:56.093899 14654 catalog_manager.cc:5495] T d396cddfe7674886b5ba7519d8495bf3 P 07ae11c0def5467ea650240db0071d78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07ae11c0def5467ea650240db0071d78 (127.5.77.194). New cstate: current_term: 1 leader_uuid: "07ae11c0def5467ea650240db0071d78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } health_report { overall_health: UNKNOWN } } }
I20260924 12:48:56.103705 14901 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e2027d642db4900bea24b47baf00757 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:52712
I20260924 12:48:56.104553 15177 tablet_replica.cc:330] T 6e2027d642db4900bea24b47baf00757 P 07ae11c0def5467ea650240db0071d78: stopping tablet replica
I20260924 12:48:56.105039 15177 raft_consensus.cc:2229] T 6e2027d642db4900bea24b47baf00757 P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:48:56.105581 15177 pending_rounds.cc:62] T 6e2027d642db4900bea24b47baf00757 P 07ae11c0def5467ea650240db0071d78: Trying to abort 1 pending ops.
I20260924 12:48:56.105716 15177 pending_rounds.cc:68] T 6e2027d642db4900bea24b47baf00757 P 07ae11c0def5467ea650240db0071d78: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332880940711899136 op_type: NO_OP noop_request { }
I20260924 12:48:56.105973 15177 raft_consensus.cc:2866] T 6e2027d642db4900bea24b47baf00757 P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260924 12:48:56.106096 15177 raft_consensus.cc:2258] T 6e2027d642db4900bea24b47baf00757 P 07ae11c0def5467ea650240db0071d78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:48:56.107759 15177 ts_tablet_manager.cc:1905] T 6e2027d642db4900bea24b47baf00757 P 07ae11c0def5467ea650240db0071d78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 12:48:56.112480 15177 ts_tablet_manager.cc:1918] T 6e2027d642db4900bea24b47baf00757 P 07ae11c0def5467ea650240db0071d78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260924 12:48:56.112674 15177 log.cc:1196] T 6e2027d642db4900bea24b47baf00757 P 07ae11c0def5467ea650240db0071d78: Deleting WAL directory at /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/wals/6e2027d642db4900bea24b47baf00757
I20260924 12:48:56.122901 14901 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a500eb27ec84386ac2e26638a9c4221 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:52712
I20260924 12:48:56.123185 15177 tablet_replica.cc:330] T 2a500eb27ec84386ac2e26638a9c4221 P 07ae11c0def5467ea650240db0071d78: stopping tablet replica
I20260924 12:48:56.123363 15177 raft_consensus.cc:2229] T 2a500eb27ec84386ac2e26638a9c4221 P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:48:56.123565 15177 pending_rounds.cc:62] T 2a500eb27ec84386ac2e26638a9c4221 P 07ae11c0def5467ea650240db0071d78: Trying to abort 1 pending ops.
I20260924 12:48:56.123627 15177 pending_rounds.cc:68] T 2a500eb27ec84386ac2e26638a9c4221 P 07ae11c0def5467ea650240db0071d78: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332880940796989440 op_type: NO_OP noop_request { }
I20260924 12:48:56.123739 15177 raft_consensus.cc:2866] T 2a500eb27ec84386ac2e26638a9c4221 P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260924 12:48:56.123828 15177 raft_consensus.cc:2258] T 2a500eb27ec84386ac2e26638a9c4221 P 07ae11c0def5467ea650240db0071d78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:48:56.124641 15177 ts_tablet_manager.cc:1905] T 2a500eb27ec84386ac2e26638a9c4221 P 07ae11c0def5467ea650240db0071d78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 12:48:56.128756 15177 ts_tablet_manager.cc:1918] T 2a500eb27ec84386ac2e26638a9c4221 P 07ae11c0def5467ea650240db0071d78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260924 12:48:56.128868 15177 log.cc:1196] T 2a500eb27ec84386ac2e26638a9c4221 P 07ae11c0def5467ea650240db0071d78: Deleting WAL directory at /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/wals/2a500eb27ec84386ac2e26638a9c4221
I20260924 12:48:56.138734 14901 tablet_service.cc:1514] Processing DeleteTablet for tablet e9f03002a94945c692b7b26f332d95e8 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:52712
I20260924 12:48:56.139012 15177 tablet_replica.cc:330] T e9f03002a94945c692b7b26f332d95e8 P 07ae11c0def5467ea650240db0071d78: stopping tablet replica
I20260924 12:48:56.139173 15177 raft_consensus.cc:2229] T e9f03002a94945c692b7b26f332d95e8 P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:48:56.139387 15177 pending_rounds.cc:62] T e9f03002a94945c692b7b26f332d95e8 P 07ae11c0def5467ea650240db0071d78: Trying to abort 1 pending ops.
I20260924 12:48:56.139432 15177 pending_rounds.cc:68] T e9f03002a94945c692b7b26f332d95e8 P 07ae11c0def5467ea650240db0071d78: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332880940875886592 op_type: NO_OP noop_request { }
I20260924 12:48:56.139549 15177 raft_consensus.cc:2866] T e9f03002a94945c692b7b26f332d95e8 P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260924 12:48:56.139626 15177 raft_consensus.cc:2258] T e9f03002a94945c692b7b26f332d95e8 P 07ae11c0def5467ea650240db0071d78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:48:56.140419 15177 ts_tablet_manager.cc:1905] T e9f03002a94945c692b7b26f332d95e8 P 07ae11c0def5467ea650240db0071d78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 12:48:56.144459 15177 ts_tablet_manager.cc:1918] T e9f03002a94945c692b7b26f332d95e8 P 07ae11c0def5467ea650240db0071d78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260924 12:48:56.144603 15177 log.cc:1196] T e9f03002a94945c692b7b26f332d95e8 P 07ae11c0def5467ea650240db0071d78: Deleting WAL directory at /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/wals/e9f03002a94945c692b7b26f332d95e8
I20260924 12:48:56.154213 14901 tablet_service.cc:1514] Processing DeleteTablet for tablet 523d9fb810d440d5ba1b16fd3da57602 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:52712
I20260924 12:48:56.154542 15177 tablet_replica.cc:330] T 523d9fb810d440d5ba1b16fd3da57602 P 07ae11c0def5467ea650240db0071d78: stopping tablet replica
I20260924 12:48:56.154744 15177 raft_consensus.cc:2229] T 523d9fb810d440d5ba1b16fd3da57602 P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:48:56.154951 15177 pending_rounds.cc:62] T 523d9fb810d440d5ba1b16fd3da57602 P 07ae11c0def5467ea650240db0071d78: Trying to abort 1 pending ops.
I20260924 12:48:56.155012 15177 pending_rounds.cc:68] T 523d9fb810d440d5ba1b16fd3da57602 P 07ae11c0def5467ea650240db0071d78: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332880940948631552 op_type: NO_OP noop_request { }
I20260924 12:48:56.155212 15177 raft_consensus.cc:2866] T 523d9fb810d440d5ba1b16fd3da57602 P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260924 12:48:56.155356 15177 raft_consensus.cc:2258] T 523d9fb810d440d5ba1b16fd3da57602 P 07ae11c0def5467ea650240db0071d78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:48:56.156162 15177 ts_tablet_manager.cc:1905] T 523d9fb810d440d5ba1b16fd3da57602 P 07ae11c0def5467ea650240db0071d78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 12:48:56.160151 15177 ts_tablet_manager.cc:1918] T 523d9fb810d440d5ba1b16fd3da57602 P 07ae11c0def5467ea650240db0071d78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260924 12:48:56.160261 15177 log.cc:1196] T 523d9fb810d440d5ba1b16fd3da57602 P 07ae11c0def5467ea650240db0071d78: Deleting WAL directory at /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/wals/523d9fb810d440d5ba1b16fd3da57602
I20260924 12:48:56.170204 14901 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c66212111554f04893157d46ea41a71 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:52712
I20260924 12:48:56.170539 15177 tablet_replica.cc:330] T 5c66212111554f04893157d46ea41a71 P 07ae11c0def5467ea650240db0071d78: stopping tablet replica
I20260924 12:48:56.170760 15177 raft_consensus.cc:2229] T 5c66212111554f04893157d46ea41a71 P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:48:56.171030 15177 pending_rounds.cc:62] T 5c66212111554f04893157d46ea41a71 P 07ae11c0def5467ea650240db0071d78: Trying to abort 1 pending ops.
I20260924 12:48:56.171084 15177 pending_rounds.cc:68] T 5c66212111554f04893157d46ea41a71 P 07ae11c0def5467ea650240db0071d78: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332880941021696000 op_type: NO_OP noop_request { }
I20260924 12:48:56.171232 15177 raft_consensus.cc:2866] T 5c66212111554f04893157d46ea41a71 P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260924 12:48:56.171334 15177 raft_consensus.cc:2258] T 5c66212111554f04893157d46ea41a71 P 07ae11c0def5467ea650240db0071d78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:48:56.172163 15177 ts_tablet_manager.cc:1905] T 5c66212111554f04893157d46ea41a71 P 07ae11c0def5467ea650240db0071d78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 12:48:56.176265 15177 ts_tablet_manager.cc:1918] T 5c66212111554f04893157d46ea41a71 P 07ae11c0def5467ea650240db0071d78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260924 12:48:56.176388 15177 log.cc:1196] T 5c66212111554f04893157d46ea41a71 P 07ae11c0def5467ea650240db0071d78: Deleting WAL directory at /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/wals/5c66212111554f04893157d46ea41a71
I20260924 12:48:56.186058 14928 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d011c630724442dfab74007165a31164"
dest_uuid: "07ae11c0def5467ea650240db0071d78"
from {username='slave'} at 127.0.0.1:52712
I20260924 12:48:56.186317 14928 raft_consensus.cc:580] T d011c630724442dfab74007165a31164 P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: Received request to step down
I20260924 12:48:56.186419 14928 raft_consensus.cc:3032] T d011c630724442dfab74007165a31164 P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 12:48:56.186502 14928 raft_consensus.cc:729] T d011c630724442dfab74007165a31164 P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 07ae11c0def5467ea650240db0071d78, State: Running, Role: LEADER
I20260924 12:48:56.186766 14928 consensus_queue.cc:260] T d011c630724442dfab74007165a31164 P 07ae11c0def5467ea650240db0071d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:56.187037 14928 raft_consensus.cc:3037] T d011c630724442dfab74007165a31164 P 07ae11c0def5467ea650240db0071d78 [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:48:56.196372 14901 tablet_service.cc:1514] Processing DeleteTablet for tablet d011c630724442dfab74007165a31164 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:52712
I20260924 12:48:56.196655 15177 tablet_replica.cc:330] T d011c630724442dfab74007165a31164 P 07ae11c0def5467ea650240db0071d78: stopping tablet replica
I20260924 12:48:56.196859 15177 raft_consensus.cc:2229] T d011c630724442dfab74007165a31164 P 07ae11c0def5467ea650240db0071d78 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260924 12:48:56.197019 15177 pending_rounds.cc:62] T d011c630724442dfab74007165a31164 P 07ae11c0def5467ea650240db0071d78: Trying to abort 1 pending ops.
I20260924 12:48:56.197088 15177 pending_rounds.cc:68] T d011c630724442dfab74007165a31164 P 07ae11c0def5467ea650240db0071d78: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332880941103325184 op_type: NO_OP noop_request { }
I20260924 12:48:56.197242 15177 raft_consensus.cc:2866] T d011c630724442dfab74007165a31164 P 07ae11c0def5467ea650240db0071d78 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260924 12:48:56.197364 15177 raft_consensus.cc:2258] T d011c630724442dfab74007165a31164 P 07ae11c0def5467ea650240db0071d78 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260924 12:48:56.198185 15177 ts_tablet_manager.cc:1905] T d011c630724442dfab74007165a31164 P 07ae11c0def5467ea650240db0071d78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 12:48:56.202262 15177 ts_tablet_manager.cc:1918] T d011c630724442dfab74007165a31164 P 07ae11c0def5467ea650240db0071d78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260924 12:48:56.202386 15177 log.cc:1196] T d011c630724442dfab74007165a31164 P 07ae11c0def5467ea650240db0071d78: Deleting WAL directory at /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/wals/d011c630724442dfab74007165a31164
I20260924 12:48:56.211871 14928 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3dd93acfeb704bc6b84977a1f6946aa3"
dest_uuid: "07ae11c0def5467ea650240db0071d78"
from {username='slave'} at 127.0.0.1:52712
I20260924 12:48:56.212090 14928 raft_consensus.cc:580] T 3dd93acfeb704bc6b84977a1f6946aa3 P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: Received request to step down
I20260924 12:48:56.212188 14928 raft_consensus.cc:3032] T 3dd93acfeb704bc6b84977a1f6946aa3 P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 12:48:56.212252 14928 raft_consensus.cc:729] T 3dd93acfeb704bc6b84977a1f6946aa3 P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 07ae11c0def5467ea650240db0071d78, State: Running, Role: LEADER
I20260924 12:48:56.212473 14928 consensus_queue.cc:260] T 3dd93acfeb704bc6b84977a1f6946aa3 P 07ae11c0def5467ea650240db0071d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:56.212764 14928 raft_consensus.cc:3037] T 3dd93acfeb704bc6b84977a1f6946aa3 P 07ae11c0def5467ea650240db0071d78 [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:48:56.221944 14901 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dd93acfeb704bc6b84977a1f6946aa3 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:52712
I20260924 12:48:56.222252 15177 tablet_replica.cc:330] T 3dd93acfeb704bc6b84977a1f6946aa3 P 07ae11c0def5467ea650240db0071d78: stopping tablet replica
I20260924 12:48:56.222402 15177 raft_consensus.cc:2229] T 3dd93acfeb704bc6b84977a1f6946aa3 P 07ae11c0def5467ea650240db0071d78 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260924 12:48:56.222613 15177 pending_rounds.cc:62] T 3dd93acfeb704bc6b84977a1f6946aa3 P 07ae11c0def5467ea650240db0071d78: Trying to abort 1 pending ops.
I20260924 12:48:56.222668 15177 pending_rounds.cc:68] T 3dd93acfeb704bc6b84977a1f6946aa3 P 07ae11c0def5467ea650240db0071d78: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332880941182062592 op_type: NO_OP noop_request { }
I20260924 12:48:56.222818 15177 raft_consensus.cc:2866] T 3dd93acfeb704bc6b84977a1f6946aa3 P 07ae11c0def5467ea650240db0071d78 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260924 12:48:56.222934 15177 raft_consensus.cc:2258] T 3dd93acfeb704bc6b84977a1f6946aa3 P 07ae11c0def5467ea650240db0071d78 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260924 12:48:56.223740 15177 ts_tablet_manager.cc:1905] T 3dd93acfeb704bc6b84977a1f6946aa3 P 07ae11c0def5467ea650240db0071d78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 12:48:56.227906 15177 ts_tablet_manager.cc:1918] T 3dd93acfeb704bc6b84977a1f6946aa3 P 07ae11c0def5467ea650240db0071d78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260924 12:48:56.228039 15177 log.cc:1196] T 3dd93acfeb704bc6b84977a1f6946aa3 P 07ae11c0def5467ea650240db0071d78: Deleting WAL directory at /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/wals/3dd93acfeb704bc6b84977a1f6946aa3
I20260924 12:48:56.237553 14928 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9723e825f07d48c9bc9fd927dd8ce48e"
dest_uuid: "07ae11c0def5467ea650240db0071d78"
from {username='slave'} at 127.0.0.1:52712
I20260924 12:48:56.237776 14928 raft_consensus.cc:580] T 9723e825f07d48c9bc9fd927dd8ce48e P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: Received request to step down
I20260924 12:48:56.237865 14928 raft_consensus.cc:3032] T 9723e825f07d48c9bc9fd927dd8ce48e P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 12:48:56.237918 14928 raft_consensus.cc:729] T 9723e825f07d48c9bc9fd927dd8ce48e P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 07ae11c0def5467ea650240db0071d78, State: Running, Role: LEADER
I20260924 12:48:56.238093 14928 consensus_queue.cc:260] T 9723e825f07d48c9bc9fd927dd8ce48e P 07ae11c0def5467ea650240db0071d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:56.238322 14928 raft_consensus.cc:3037] T 9723e825f07d48c9bc9fd927dd8ce48e P 07ae11c0def5467ea650240db0071d78 [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:48:56.247869 14901 tablet_service.cc:1514] Processing DeleteTablet for tablet 9723e825f07d48c9bc9fd927dd8ce48e with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:52712
I20260924 12:48:56.248240 15177 tablet_replica.cc:330] T 9723e825f07d48c9bc9fd927dd8ce48e P 07ae11c0def5467ea650240db0071d78: stopping tablet replica
I20260924 12:48:56.248445 15177 raft_consensus.cc:2229] T 9723e825f07d48c9bc9fd927dd8ce48e P 07ae11c0def5467ea650240db0071d78 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260924 12:48:56.248629 15177 pending_rounds.cc:62] T 9723e825f07d48c9bc9fd927dd8ce48e P 07ae11c0def5467ea650240db0071d78: Trying to abort 1 pending ops.
I20260924 12:48:56.248709 15177 pending_rounds.cc:68] T 9723e825f07d48c9bc9fd927dd8ce48e P 07ae11c0def5467ea650240db0071d78: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332880941259907072 op_type: NO_OP noop_request { }
I20260924 12:48:56.248839 15177 raft_consensus.cc:2866] T 9723e825f07d48c9bc9fd927dd8ce48e P 07ae11c0def5467ea650240db0071d78 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260924 12:48:56.248955 15177 raft_consensus.cc:2258] T 9723e825f07d48c9bc9fd927dd8ce48e P 07ae11c0def5467ea650240db0071d78 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260924 12:48:56.249626 15177 ts_tablet_manager.cc:1905] T 9723e825f07d48c9bc9fd927dd8ce48e P 07ae11c0def5467ea650240db0071d78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 12:48:56.254011 15177 ts_tablet_manager.cc:1918] T 9723e825f07d48c9bc9fd927dd8ce48e P 07ae11c0def5467ea650240db0071d78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260924 12:48:56.254112 15177 log.cc:1196] T 9723e825f07d48c9bc9fd927dd8ce48e P 07ae11c0def5467ea650240db0071d78: Deleting WAL directory at /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/wals/9723e825f07d48c9bc9fd927dd8ce48e
I20260924 12:48:56.263710 14928 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "75c3b8a29aa1489da08a21b6cf353bc6"
dest_uuid: "07ae11c0def5467ea650240db0071d78"
from {username='slave'} at 127.0.0.1:52712
I20260924 12:48:56.263911 14928 raft_consensus.cc:580] T 75c3b8a29aa1489da08a21b6cf353bc6 P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: Received request to step down
I20260924 12:48:56.263993 14928 raft_consensus.cc:3032] T 75c3b8a29aa1489da08a21b6cf353bc6 P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 12:48:56.264040 14928 raft_consensus.cc:729] T 75c3b8a29aa1489da08a21b6cf353bc6 P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 07ae11c0def5467ea650240db0071d78, State: Running, Role: LEADER
I20260924 12:48:56.264226 14928 consensus_queue.cc:260] T 75c3b8a29aa1489da08a21b6cf353bc6 P 07ae11c0def5467ea650240db0071d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:56.264447 14928 raft_consensus.cc:3037] T 75c3b8a29aa1489da08a21b6cf353bc6 P 07ae11c0def5467ea650240db0071d78 [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:48:56.273706 14901 tablet_service.cc:1514] Processing DeleteTablet for tablet 75c3b8a29aa1489da08a21b6cf353bc6 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:52712
I20260924 12:48:56.274065 15177 tablet_replica.cc:330] T 75c3b8a29aa1489da08a21b6cf353bc6 P 07ae11c0def5467ea650240db0071d78: stopping tablet replica
I20260924 12:48:56.274276 15177 raft_consensus.cc:2229] T 75c3b8a29aa1489da08a21b6cf353bc6 P 07ae11c0def5467ea650240db0071d78 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260924 12:48:56.274501 15177 pending_rounds.cc:62] T 75c3b8a29aa1489da08a21b6cf353bc6 P 07ae11c0def5467ea650240db0071d78: Trying to abort 1 pending ops.
I20260924 12:48:56.274565 15177 pending_rounds.cc:68] T 75c3b8a29aa1489da08a21b6cf353bc6 P 07ae11c0def5467ea650240db0071d78: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332880941341298688 op_type: NO_OP noop_request { }
I20260924 12:48:56.274760 15177 raft_consensus.cc:2866] T 75c3b8a29aa1489da08a21b6cf353bc6 P 07ae11c0def5467ea650240db0071d78 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260924 12:48:56.274889 15177 raft_consensus.cc:2258] T 75c3b8a29aa1489da08a21b6cf353bc6 P 07ae11c0def5467ea650240db0071d78 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260924 12:48:56.275668 15177 ts_tablet_manager.cc:1905] T 75c3b8a29aa1489da08a21b6cf353bc6 P 07ae11c0def5467ea650240db0071d78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 12:48:56.280097 15177 ts_tablet_manager.cc:1918] T 75c3b8a29aa1489da08a21b6cf353bc6 P 07ae11c0def5467ea650240db0071d78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260924 12:48:56.280233 15177 log.cc:1196] T 75c3b8a29aa1489da08a21b6cf353bc6 P 07ae11c0def5467ea650240db0071d78: Deleting WAL directory at /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/wals/75c3b8a29aa1489da08a21b6cf353bc6
I20260924 12:48:56.289907 14928 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d396cddfe7674886b5ba7519d8495bf3"
dest_uuid: "07ae11c0def5467ea650240db0071d78"
from {username='slave'} at 127.0.0.1:52712
I20260924 12:48:56.290112 14928 raft_consensus.cc:580] T d396cddfe7674886b5ba7519d8495bf3 P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: Received request to step down
I20260924 12:48:56.290205 14928 raft_consensus.cc:3032] T d396cddfe7674886b5ba7519d8495bf3 P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 12:48:56.290252 14928 raft_consensus.cc:729] T d396cddfe7674886b5ba7519d8495bf3 P 07ae11c0def5467ea650240db0071d78 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 07ae11c0def5467ea650240db0071d78, State: Running, Role: LEADER
I20260924 12:48:56.290426 14928 consensus_queue.cc:260] T d396cddfe7674886b5ba7519d8495bf3 P 07ae11c0def5467ea650240db0071d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16464c128b8649d6bafdf4f45f981031" member_type: VOTER last_known_addr { host: "127.5.77.193" port: 37587 } } peers { permanent_uuid: "07ae11c0def5467ea650240db0071d78" member_type: VOTER last_known_addr { host: "127.5.77.194" port: 34953 } } peers { permanent_uuid: "af5894d0b4fe4f3ea1ebb39472de0eb7" member_type: VOTER last_known_addr { host: "127.5.77.195" port: 38363 } }
I20260924 12:48:56.290693 14928 raft_consensus.cc:3037] T d396cddfe7674886b5ba7519d8495bf3 P 07ae11c0def5467ea650240db0071d78 [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:48:56.300442 14901 tablet_service.cc:1514] Processing DeleteTablet for tablet d396cddfe7674886b5ba7519d8495bf3 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:52712
I20260924 12:48:56.300868 15177 tablet_replica.cc:330] T d396cddfe7674886b5ba7519d8495bf3 P 07ae11c0def5467ea650240db0071d78: stopping tablet replica
I20260924 12:48:56.301050 15177 raft_consensus.cc:2229] T d396cddfe7674886b5ba7519d8495bf3 P 07ae11c0def5467ea650240db0071d78 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260924 12:48:56.301209 15177 pending_rounds.cc:62] T d396cddfe7674886b5ba7519d8495bf3 P 07ae11c0def5467ea650240db0071d78: Trying to abort 1 pending ops.
I20260924 12:48:56.301275 15177 pending_rounds.cc:68] T d396cddfe7674886b5ba7519d8495bf3 P 07ae11c0def5467ea650240db0071d78: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332880941430517760 op_type: NO_OP noop_request { }
I20260924 12:48:56.301414 15177 raft_consensus.cc:2866] T d396cddfe7674886b5ba7519d8495bf3 P 07ae11c0def5467ea650240db0071d78 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260924 12:48:56.301530 15177 raft_consensus.cc:2258] T d396cddfe7674886b5ba7519d8495bf3 P 07ae11c0def5467ea650240db0071d78 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260924 12:48:56.302199 15177 ts_tablet_manager.cc:1905] T d396cddfe7674886b5ba7519d8495bf3 P 07ae11c0def5467ea650240db0071d78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 12:48:56.306234 15177 ts_tablet_manager.cc:1918] T d396cddfe7674886b5ba7519d8495bf3 P 07ae11c0def5467ea650240db0071d78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260924 12:48:56.306337 15177 log.cc:1196] T d396cddfe7674886b5ba7519d8495bf3 P 07ae11c0def5467ea650240db0071d78: Deleting WAL directory at /tmp/dist-test-taskL2kveR/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790253941578571-5431-0/raft_consensus-itest-cluster/ts-1/wal/wals/d396cddfe7674886b5ba7519d8495bf3
I20260924 12:48:56.318185 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 14715
I20260924 12:48:56.324467 14837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:56.418612 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 14715
I20260924 12:48:56.432808 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 14846
I20260924 12:48:56.438977 14968 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:56.548144 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 14846
I20260924 12:48:56.562824 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 14976
I20260924 12:48:56.568878 15098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:56.664700 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 14976
I20260924 12:48:56.677788 5431 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskL2kveR/build/asan/bin/kudu pid 14625
I20260924 12:48:56.678920 14686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:56.783409 5431 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL2kveR/build/asan/bin/kudu with pid 14625
2026-09-24T12:48:56Z chronyd exiting
[ OK ] NumReplicas/RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric/1 (3155 ms)
[----------] 2 tests from NumReplicas/RaftConsensusNumLeadersMetricTest (5001 ms total)
[----------] Global test environment tear-down
[==========] 14 tests from 2 test suites ran. (195173 ms total)
[ PASSED ] 13 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] RaftConsensusElectionITest.TestNewLeaderCantResolvePeers
1 FAILED TEST
I20260924 12:48:56.824921 5431 logging.cc:424] LogThrottler /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 46 messages since previous log ~15 seconds ago