[==========] Running 14 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 12 tests from RaftConsensusElectionITest
[ RUN ] RaftConsensusElectionITest.TestNewLeaderCantResolvePeers
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260804 14:50:48.140779 24602 test_util.cc:268] Using random seed: -943470671
I20260804 14:50:48.152573 24602 ts_itest-base.cc:112] Starting cluster with:
I20260804 14:50:48.152722 24602 ts_itest-base.cc:113] --------------
I20260804 14:50:48.152745 24602 ts_itest-base.cc:114] 3 tablet servers
I20260804 14:50:48.152787 24602 ts_itest-base.cc:115] 3 replicas per TS
I20260804 14:50:48.152815 24602 ts_itest-base.cc:116] --------------
2026-08-04T14:50:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-04T14:50:48Z Disabled control of system clock
I20260804 14:50:48.170562 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.6.190:39931
--webserver_interface=127.24.6.190
--webserver_port=0
--builtin_ntp_servers=127.24.6.148:44795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.6.190:39931
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20260804 14:50:48.302596 24617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:50:48.303011 24617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:50:48.303098 24617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:50:48.307574 24617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260804 14:50:48.307693 24617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:50:48.307726 24617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260804 14:50:48.307768 24617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260804 14:50:48.313211 24617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:44795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.24.6.190:39931
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.6.190:39931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/master-0/data/info.pb
--webserver_interface=127.24.6.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.24617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:50:48.315001 24617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:50:48.317302 24617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260804 14:50:48.330834 24617 server_base.cc:1034] running on GCE node
W20260804 14:50:48.330977 24626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:50:48.330838 24627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:50:48.331314 24631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:50:48.332136 24617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:50:48.333956 24617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:50:48.335179 24617 hybrid_clock.cc:648] HybridClock initialized: now 1785855048335133 us; error 69 us; skew 500 ppm
I20260804 14:50:48.338129 24617 webserver.cc:460] Webserver started at http://127.24.6.190:38323/ using document root <none> and password file <none>
I20260804 14:50:48.339082 24617 fs_manager.cc:359] Metadata directory not provided
I20260804 14:50:48.339190 24617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:50:48.339600 24617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:50:48.342324 24617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/master-0/data/instance:
uuid: "2b05367e778147d4bd90a1ee043ebf32"
format_stamp: "Formatted at 2026-08-04 14:50:48 on dist-test-slave-92m1"
I20260804 14:50:48.342976 24617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/master-0/wal/instance:
uuid: "2b05367e778147d4bd90a1ee043ebf32"
format_stamp: "Formatted at 2026-08-04 14:50:48 on dist-test-slave-92m1"
I20260804 14:50:48.348708 24617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260804 14:50:48.353089 24637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:50:48.355170 24617 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260804 14:50:48.355485 24617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/master-0/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/master-0/wal
uuid: "2b05367e778147d4bd90a1ee043ebf32"
format_stamp: "Formatted at 2026-08-04 14:50:48 on dist-test-slave-92m1"
I20260804 14:50:48.355647 24617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:50:48.372659 24617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:50:48.373600 24617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:50:48.373868 24617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:50:48.383625 24617 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.190:39931
I20260804 14:50:48.383685 24715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.190:39931 every 8 connection(s)
I20260804 14:50:48.384938 24617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/master-0/data/info.pb
I20260804 14:50:48.389456 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 24617
I20260804 14:50:48.389652 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/master-0/wal/instance
I20260804 14:50:48.390468 24717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:50:48.399524 24717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b05367e778147d4bd90a1ee043ebf32: Bootstrap starting.
I20260804 14:50:48.403290 24717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b05367e778147d4bd90a1ee043ebf32: Neither blocks nor log segments found. Creating new log.
I20260804 14:50:48.404470 24717 log.cc:824] T 00000000000000000000000000000000 P 2b05367e778147d4bd90a1ee043ebf32: Log is configured to *not* fsync() on all Append() calls
I20260804 14:50:48.407279 24717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b05367e778147d4bd90a1ee043ebf32: No bootstrap required, opened a new log
I20260804 14:50:48.412688 24717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b05367e778147d4bd90a1ee043ebf32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b05367e778147d4bd90a1ee043ebf32" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 39931 } }
I20260804 14:50:48.413023 24717 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b05367e778147d4bd90a1ee043ebf32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:50:48.413146 24717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b05367e778147d4bd90a1ee043ebf32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b05367e778147d4bd90a1ee043ebf32, State: Initialized, Role: FOLLOWER
I20260804 14:50:48.413892 24717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b05367e778147d4bd90a1ee043ebf32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b05367e778147d4bd90a1ee043ebf32" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 39931 } }
I20260804 14:50:48.414114 24717 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b05367e778147d4bd90a1ee043ebf32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260804 14:50:48.414240 24717 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b05367e778147d4bd90a1ee043ebf32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260804 14:50:48.414424 24717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b05367e778147d4bd90a1ee043ebf32 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:50:48.415712 24717 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b05367e778147d4bd90a1ee043ebf32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b05367e778147d4bd90a1ee043ebf32" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 39931 } }
I20260804 14:50:48.416342 24717 leader_election.cc:302] T 00000000000000000000000000000000 P 2b05367e778147d4bd90a1ee043ebf32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b05367e778147d4bd90a1ee043ebf32; no voters:
I20260804 14:50:48.416818 24717 leader_election.cc:288] T 00000000000000000000000000000000 P 2b05367e778147d4bd90a1ee043ebf32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260804 14:50:48.416958 24723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b05367e778147d4bd90a1ee043ebf32 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:50:48.417224 24723 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b05367e778147d4bd90a1ee043ebf32 [term 1 LEADER]: Becoming Leader. State: Replica: 2b05367e778147d4bd90a1ee043ebf32, State: Running, Role: LEADER
I20260804 14:50:48.417639 24723 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b05367e778147d4bd90a1ee043ebf32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b05367e778147d4bd90a1ee043ebf32" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 39931 } }
I20260804 14:50:48.418486 24717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b05367e778147d4bd90a1ee043ebf32 [sys.catalog]: configured and running, proceeding with master startup.
I20260804 14:50:48.420415 24724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b05367e778147d4bd90a1ee043ebf32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b05367e778147d4bd90a1ee043ebf32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b05367e778147d4bd90a1ee043ebf32" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 39931 } } }
I20260804 14:50:48.420557 24724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b05367e778147d4bd90a1ee043ebf32 [sys.catalog]: This master's current role is: LEADER
I20260804 14:50:48.420938 24726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b05367e778147d4bd90a1ee043ebf32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b05367e778147d4bd90a1ee043ebf32. Latest consensus state: current_term: 1 leader_uuid: "2b05367e778147d4bd90a1ee043ebf32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b05367e778147d4bd90a1ee043ebf32" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 39931 } } }
I20260804 14:50:48.421062 24726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b05367e778147d4bd90a1ee043ebf32 [sys.catalog]: This master's current role is: LEADER
I20260804 14:50:48.421203 24736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260804 14:50:48.425508 24736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260804 14:50:48.433578 24736 catalog_manager.cc:1347] Generated new cluster ID: 419622c07d2c404bab3732428123d1b6
I20260804 14:50:48.433668 24736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260804 14:50:48.445310 24736 catalog_manager.cc:1370] Generated new certificate authority record
I20260804 14:50:48.446908 24736 catalog_manager.cc:1504] Loading token signing keys...
I20260804 14:50:48.458189 24736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b05367e778147d4bd90a1ee043ebf32: Generated new TSK 0
I20260804 14:50:48.459448 24736 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260804 14:50:48.470598 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.129:0
--local_ip_for_outbound_sockets=127.24.6.129
--webserver_interface=127.24.6.129
--webserver_port=0
--tserver_master_addrs=127.24.6.190:39931
--builtin_ntp_servers=127.24.6.148:44795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--dns_resolver_cache_capacity_mb=0
--enable_leader_failure_detection=false
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=3
--leader_failure_max_missed_heartbeat_periods=10
--leader_failure_exp_backoff_max_delta_ms=5000 with env {}
W20260804 14:50:48.608500 24750 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260804 14:50:48.608815 24750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:50:48.608837 24750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:50:48.608912 24750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:50:48.613250 24750 flags.cc:425] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=5000
W20260804 14:50:48.613338 24750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:50:48.613397 24750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.129
I20260804 14:50:48.618953 24750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:44795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--enable_leader_failure_detection=false
--leader_failure_exp_backoff_max_delta_ms=5000
--leader_failure_max_missed_heartbeat_periods=10
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.24.6.129
--webserver_port=0
--tserver_master_addrs=127.24.6.190:39931
--never_fsync=true
--heap_profile_path=/tmp/kudu.24750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--dns_resolver_cache_capacity_mb=0
--local_ip_for_outbound_sockets=127.24.6.129
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:50:48.620451 24750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:50:48.622066 24750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260804 14:50:48.633462 24750 server_base.cc:1034] running on GCE node
W20260804 14:50:48.633453 24761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:50:48.633484 24760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:50:48.633481 24763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:50:48.634214 24750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:50:48.634950 24750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:50:48.636163 24750 hybrid_clock.cc:648] HybridClock initialized: now 1785855048636129 us; error 46 us; skew 500 ppm
I20260804 14:50:48.638209 24750 webserver.cc:460] Webserver started at http://127.24.6.129:36015/ using document root <none> and password file <none>
I20260804 14:50:48.638849 24750 fs_manager.cc:359] Metadata directory not provided
I20260804 14:50:48.638943 24750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:50:48.639263 24750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:50:48.641335 24750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-0/data/instance:
uuid: "8d570a14f7d34a5d9affae0f3108d60c"
format_stamp: "Formatted at 2026-08-04 14:50:48 on dist-test-slave-92m1"
I20260804 14:50:48.641947 24750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-0/wal/instance:
uuid: "8d570a14f7d34a5d9affae0f3108d60c"
format_stamp: "Formatted at 2026-08-04 14:50:48 on dist-test-slave-92m1"
I20260804 14:50:48.646167 24750 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260804 14:50:48.648998 24773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:50:48.650411 24750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260804 14:50:48.650559 24750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-0/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-0/wal
uuid: "8d570a14f7d34a5d9affae0f3108d60c"
format_stamp: "Formatted at 2026-08-04 14:50:48 on dist-test-slave-92m1"
I20260804 14:50:48.650682 24750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:50:48.665704 24750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:50:48.666493 24750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:50:48.666723 24750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:50:48.667593 24750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:50:48.669037 24750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:50:48.669121 24750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:50:48.669198 24750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:50:48.669255 24750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260804 14:50:48.679787 24750 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.129:44825
I20260804 14:50:48.679819 24930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.129:44825 every 8 connection(s)
I20260804 14:50:48.680764 24750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-0/data/info.pb
I20260804 14:50:48.689867 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 24750
I20260804 14:50:48.690022 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-0/wal/instance
I20260804 14:50:48.692659 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.130:0
--local_ip_for_outbound_sockets=127.24.6.130
--webserver_interface=127.24.6.130
--webserver_port=0
--tserver_master_addrs=127.24.6.190:39931
--builtin_ntp_servers=127.24.6.148:44795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--dns_resolver_cache_capacity_mb=0
--enable_leader_failure_detection=false
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=3
--leader_failure_max_missed_heartbeat_periods=10
--leader_failure_exp_backoff_max_delta_ms=5000 with env {}
I20260804 14:50:48.693277 24931 heartbeater.cc:347] Connected to a master server at 127.24.6.190:39931
I20260804 14:50:48.693570 24931 heartbeater.cc:464] Registering TS with master...
I20260804 14:50:48.694304 24931 heartbeater.cc:511] Master 127.24.6.190:39931 requested a full tablet report, sending...
I20260804 14:50:48.697069 24665 ts_manager.cc:194] Registered new tserver with Master: 8d570a14f7d34a5d9affae0f3108d60c (127.24.6.129:44825)
I20260804 14:50:48.698671 24665 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.129:47229
W20260804 14:50:48.819279 24934 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260804 14:50:48.819623 24934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:50:48.819646 24934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:50:48.819720 24934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:50:48.824335 24934 flags.cc:425] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=5000
W20260804 14:50:48.824429 24934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:50:48.824487 24934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.130
I20260804 14:50:48.829583 24934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:44795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--enable_leader_failure_detection=false
--leader_failure_exp_backoff_max_delta_ms=5000
--leader_failure_max_missed_heartbeat_periods=10
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.24.6.130
--webserver_port=0
--tserver_master_addrs=127.24.6.190:39931
--never_fsync=true
--heap_profile_path=/tmp/kudu.24934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--dns_resolver_cache_capacity_mb=0
--local_ip_for_outbound_sockets=127.24.6.130
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:50:48.830883 24934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:50:48.832563 24934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:50:48.843730 24941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:50:48.843757 24940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:50:48.843973 24943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:50:48.844107 24934 server_base.cc:1034] running on GCE node
I20260804 14:50:48.844592 24934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:50:48.845376 24934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:50:48.846575 24934 hybrid_clock.cc:648] HybridClock initialized: now 1785855048846541 us; error 45 us; skew 500 ppm
I20260804 14:50:48.848837 24934 webserver.cc:460] Webserver started at http://127.24.6.130:40809/ using document root <none> and password file <none>
I20260804 14:50:48.849572 24934 fs_manager.cc:359] Metadata directory not provided
I20260804 14:50:48.849646 24934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:50:48.849925 24934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:50:48.852288 24934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-1/data/instance:
uuid: "4ae609bd205d473797afce12a389988d"
format_stamp: "Formatted at 2026-08-04 14:50:48 on dist-test-slave-92m1"
I20260804 14:50:48.852948 24934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-1/wal/instance:
uuid: "4ae609bd205d473797afce12a389988d"
format_stamp: "Formatted at 2026-08-04 14:50:48 on dist-test-slave-92m1"
I20260804 14:50:48.857352 24934 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260804 14:50:48.860253 24952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:50:48.861657 24934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260804 14:50:48.861788 24934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-1/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-1/wal
uuid: "4ae609bd205d473797afce12a389988d"
format_stamp: "Formatted at 2026-08-04 14:50:48 on dist-test-slave-92m1"
I20260804 14:50:48.861902 24934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:50:48.875826 24934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:50:48.876718 24934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:50:48.876977 24934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:50:48.877820 24934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:50:48.878989 24934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:50:48.879077 24934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:50:48.879189 24934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:50:48.879272 24934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260804 14:50:48.890527 24934 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.130:39519
I20260804 14:50:48.890551 25106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.130:39519 every 8 connection(s)
I20260804 14:50:48.891729 24934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-1/data/info.pb
I20260804 14:50:48.892198 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 24934
I20260804 14:50:48.892339 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-1/wal/instance
I20260804 14:50:48.895434 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.131:0
--local_ip_for_outbound_sockets=127.24.6.131
--webserver_interface=127.24.6.131
--webserver_port=0
--tserver_master_addrs=127.24.6.190:39931
--builtin_ntp_servers=127.24.6.148:44795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--dns_resolver_cache_capacity_mb=0
--enable_leader_failure_detection=false
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=3
--leader_failure_max_missed_heartbeat_periods=10
--leader_failure_exp_backoff_max_delta_ms=5000 with env {}
I20260804 14:50:48.906804 25109 heartbeater.cc:347] Connected to a master server at 127.24.6.190:39931
I20260804 14:50:48.907243 25109 heartbeater.cc:464] Registering TS with master...
I20260804 14:50:48.907938 25109 heartbeater.cc:511] Master 127.24.6.190:39931 requested a full tablet report, sending...
I20260804 14:50:48.909366 24665 ts_manager.cc:194] Registered new tserver with Master: 4ae609bd205d473797afce12a389988d (127.24.6.130:39519)
I20260804 14:50:48.910053 24665 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.130:56255
W20260804 14:50:49.022997 25112 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260804 14:50:49.023447 25112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:50:49.023519 25112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:50:49.023591 25112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:50:49.027853 25112 flags.cc:425] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=5000
W20260804 14:50:49.027976 25112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:50:49.028072 25112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.131
I20260804 14:50:49.033262 25112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:44795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--enable_leader_failure_detection=false
--leader_failure_exp_backoff_max_delta_ms=5000
--leader_failure_max_missed_heartbeat_periods=10
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.24.6.131
--webserver_port=0
--tserver_master_addrs=127.24.6.190:39931
--never_fsync=true
--heap_profile_path=/tmp/kudu.25112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--dns_resolver_cache_capacity_mb=0
--local_ip_for_outbound_sockets=127.24.6.131
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:50:49.034533 25112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:50:49.036038 25112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260804 14:50:49.046216 25112 server_base.cc:1034] running on GCE node
W20260804 14:50:49.046288 25119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:50:49.046248 25122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:50:49.046236 25120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:50:49.046943 25112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:50:49.047719 25112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:50:49.048914 25112 hybrid_clock.cc:648] HybridClock initialized: now 1785855049048885 us; error 41 us; skew 500 ppm
I20260804 14:50:49.051036 25112 webserver.cc:460] Webserver started at http://127.24.6.131:33621/ using document root <none> and password file <none>
I20260804 14:50:49.051718 25112 fs_manager.cc:359] Metadata directory not provided
I20260804 14:50:49.051813 25112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:50:49.052083 25112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:50:49.054200 25112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-2/data/instance:
uuid: "2f2a7b98753c479e9ce8d9db9d6490b0"
format_stamp: "Formatted at 2026-08-04 14:50:49 on dist-test-slave-92m1"
I20260804 14:50:49.054836 25112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-2/wal/instance:
uuid: "2f2a7b98753c479e9ce8d9db9d6490b0"
format_stamp: "Formatted at 2026-08-04 14:50:49 on dist-test-slave-92m1"
I20260804 14:50:49.058871 25112 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260804 14:50:49.061662 25128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:50:49.063350 25112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260804 14:50:49.063530 25112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-2/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-2/wal
uuid: "2f2a7b98753c479e9ce8d9db9d6490b0"
format_stamp: "Formatted at 2026-08-04 14:50:49 on dist-test-slave-92m1"
I20260804 14:50:49.063666 25112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:50:49.078598 25112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:50:49.079514 25112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:50:49.079748 25112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:50:49.080423 25112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:50:49.081559 25112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:50:49.081635 25112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:50:49.081718 25112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:50:49.081763 25112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:50:49.092274 25112 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.131:35841
I20260804 14:50:49.092300 25289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.131:35841 every 8 connection(s)
I20260804 14:50:49.093348 25112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-2/data/info.pb
I20260804 14:50:49.094954 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 25112
I20260804 14:50:49.095075 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-2/wal/instance
I20260804 14:50:49.106019 25290 heartbeater.cc:347] Connected to a master server at 127.24.6.190:39931
I20260804 14:50:49.106336 25290 heartbeater.cc:464] Registering TS with master...
I20260804 14:50:49.106952 25290 heartbeater.cc:511] Master 127.24.6.190:39931 requested a full tablet report, sending...
I20260804 14:50:49.108206 24665 ts_manager.cc:194] Registered new tserver with Master: 2f2a7b98753c479e9ce8d9db9d6490b0 (127.24.6.131:35841)
I20260804 14:50:49.108784 24665 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.131:51769
I20260804 14:50:49.110749 24602 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260804 14:50:49.113425 24602 test_util.cc:268] Using random seed: -942497972
I20260804 14:50:49.133579 24665 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51910:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260804 14:50:49.135179 24665 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260804 14:50:49.157596 25194 tablet_service.cc:1467] Processing CreateTablet for tablet c99d0efe445644f09815fc7c540f551d (DEFAULT_TABLE table=test-workload [id=15b6e4029344413792281b439ead2ccf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:50:49.159179 25194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c99d0efe445644f09815fc7c540f551d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:50:49.159392 24840 tablet_service.cc:1467] Processing CreateTablet for tablet c99d0efe445644f09815fc7c540f551d (DEFAULT_TABLE table=test-workload [id=15b6e4029344413792281b439ead2ccf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:50:49.159716 25021 tablet_service.cc:1467] Processing CreateTablet for tablet c99d0efe445644f09815fc7c540f551d (DEFAULT_TABLE table=test-workload [id=15b6e4029344413792281b439ead2ccf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:50:49.160565 24840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c99d0efe445644f09815fc7c540f551d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:50:49.160683 25021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c99d0efe445644f09815fc7c540f551d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:50:49.167309 25312 tablet_bootstrap.cc:492] T c99d0efe445644f09815fc7c540f551d P 4ae609bd205d473797afce12a389988d: Bootstrap starting.
I20260804 14:50:49.168601 25315 tablet_bootstrap.cc:492] T c99d0efe445644f09815fc7c540f551d P 8d570a14f7d34a5d9affae0f3108d60c: Bootstrap starting.
I20260804 14:50:49.169026 25313 tablet_bootstrap.cc:492] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0: Bootstrap starting.
I20260804 14:50:49.170516 25312 tablet_bootstrap.cc:654] T c99d0efe445644f09815fc7c540f551d P 4ae609bd205d473797afce12a389988d: Neither blocks nor log segments found. Creating new log.
I20260804 14:50:49.171192 25315 tablet_bootstrap.cc:654] T c99d0efe445644f09815fc7c540f551d P 8d570a14f7d34a5d9affae0f3108d60c: Neither blocks nor log segments found. Creating new log.
I20260804 14:50:49.171396 25313 tablet_bootstrap.cc:654] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0: Neither blocks nor log segments found. Creating new log.
I20260804 14:50:49.171386 25312 log.cc:824] T c99d0efe445644f09815fc7c540f551d P 4ae609bd205d473797afce12a389988d: Log is configured to *not* fsync() on all Append() calls
I20260804 14:50:49.172133 25315 log.cc:824] T c99d0efe445644f09815fc7c540f551d P 8d570a14f7d34a5d9affae0f3108d60c: Log is configured to *not* fsync() on all Append() calls
I20260804 14:50:49.172132 25313 log.cc:824] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0: Log is configured to *not* fsync() on all Append() calls
I20260804 14:50:49.173696 25312 tablet_bootstrap.cc:492] T c99d0efe445644f09815fc7c540f551d P 4ae609bd205d473797afce12a389988d: No bootstrap required, opened a new log
I20260804 14:50:49.173915 25312 ts_tablet_manager.cc:1397] T c99d0efe445644f09815fc7c540f551d P 4ae609bd205d473797afce12a389988d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260804 14:50:49.174132 25315 tablet_bootstrap.cc:492] T c99d0efe445644f09815fc7c540f551d P 8d570a14f7d34a5d9affae0f3108d60c: No bootstrap required, opened a new log
I20260804 14:50:49.174259 25313 tablet_bootstrap.cc:492] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0: No bootstrap required, opened a new log
I20260804 14:50:49.174381 25315 ts_tablet_manager.cc:1397] T c99d0efe445644f09815fc7c540f551d P 8d570a14f7d34a5d9affae0f3108d60c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260804 14:50:49.174441 25313 ts_tablet_manager.cc:1397] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260804 14:50:49.178433 25315 raft_consensus.cc:348] T c99d0efe445644f09815fc7c540f551d P 8d570a14f7d34a5d9affae0f3108d60c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae609bd205d473797afce12a389988d" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39519 } } peers { permanent_uuid: "2f2a7b98753c479e9ce8d9db9d6490b0" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 35841 } } peers { permanent_uuid: "8d570a14f7d34a5d9affae0f3108d60c" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44825 } }
I20260804 14:50:49.178719 25315 raft_consensus.cc:729] T c99d0efe445644f09815fc7c540f551d P 8d570a14f7d34a5d9affae0f3108d60c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d570a14f7d34a5d9affae0f3108d60c, State: Initialized, Role: FOLLOWER
I20260804 14:50:49.179056 25315 consensus_queue.cc:260] T c99d0efe445644f09815fc7c540f551d P 8d570a14f7d34a5d9affae0f3108d60c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae609bd205d473797afce12a389988d" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39519 } } peers { permanent_uuid: "2f2a7b98753c479e9ce8d9db9d6490b0" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 35841 } } peers { permanent_uuid: "8d570a14f7d34a5d9affae0f3108d60c" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44825 } }
I20260804 14:50:49.178963 25313 raft_consensus.cc:348] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae609bd205d473797afce12a389988d" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39519 } } peers { permanent_uuid: "2f2a7b98753c479e9ce8d9db9d6490b0" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 35841 } } peers { permanent_uuid: "8d570a14f7d34a5d9affae0f3108d60c" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44825 } }
I20260804 14:50:49.178962 25312 raft_consensus.cc:348] T c99d0efe445644f09815fc7c540f551d P 4ae609bd205d473797afce12a389988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae609bd205d473797afce12a389988d" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39519 } } peers { permanent_uuid: "2f2a7b98753c479e9ce8d9db9d6490b0" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 35841 } } peers { permanent_uuid: "8d570a14f7d34a5d9affae0f3108d60c" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44825 } }
I20260804 14:50:49.179239 25313 raft_consensus.cc:729] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f2a7b98753c479e9ce8d9db9d6490b0, State: Initialized, Role: FOLLOWER
I20260804 14:50:49.179239 25312 raft_consensus.cc:729] T c99d0efe445644f09815fc7c540f551d P 4ae609bd205d473797afce12a389988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ae609bd205d473797afce12a389988d, State: Initialized, Role: FOLLOWER
I20260804 14:50:49.179546 25313 consensus_queue.cc:260] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae609bd205d473797afce12a389988d" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39519 } } peers { permanent_uuid: "2f2a7b98753c479e9ce8d9db9d6490b0" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 35841 } } peers { permanent_uuid: "8d570a14f7d34a5d9affae0f3108d60c" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44825 } }
I20260804 14:50:49.179546 25312 consensus_queue.cc:260] T c99d0efe445644f09815fc7c540f551d P 4ae609bd205d473797afce12a389988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae609bd205d473797afce12a389988d" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39519 } } peers { permanent_uuid: "2f2a7b98753c479e9ce8d9db9d6490b0" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 35841 } } peers { permanent_uuid: "8d570a14f7d34a5d9affae0f3108d60c" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44825 } }
I20260804 14:50:49.180045 24931 heartbeater.cc:503] Master 127.24.6.190:39931 was elected leader, sending a full tablet report...
I20260804 14:50:49.180328 25315 ts_tablet_manager.cc:1428] T c99d0efe445644f09815fc7c540f551d P 8d570a14f7d34a5d9affae0f3108d60c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260804 14:50:49.180537 25313 ts_tablet_manager.cc:1428] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260804 14:50:49.180537 25312 ts_tablet_manager.cc:1428] T c99d0efe445644f09815fc7c540f551d P 4ae609bd205d473797afce12a389988d: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260804 14:50:49.180939 25290 heartbeater.cc:503] Master 127.24.6.190:39931 was elected leader, sending a full tablet report...
I20260804 14:50:49.181188 25109 heartbeater.cc:503] Master 127.24.6.190:39931 was elected leader, sending a full tablet report...
W20260804 14:50:49.182248 24932 tablet.cc:2369] T c99d0efe445644f09815fc7c540f551d P 8d570a14f7d34a5d9affae0f3108d60c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260804 14:50:49.223877 25100 generic_service.cc:139] {username='slave'} at 127.0.0.1:57754 forcing change of non-runtime-safe flag fail_dns_resolution_hostports
I20260804 14:50:49.224148 25100 generic_service.cc:175] {username='slave'} at 127.0.0.1:57754 changed flags via RPC: fail_dns_resolution_hostports from '' to '127.24.6.130:39519'
W20260804 14:50:49.225651 25100 generic_service.cc:139] {username='slave'} at 127.0.0.1:57754 forcing change of non-runtime-safe flag fail_dns_resolution
I20260804 14:50:49.225785 25100 generic_service.cc:175] {username='slave'} at 127.0.0.1:57754 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
W20260804 14:50:49.231326 25282 generic_service.cc:139] {username='slave'} at 127.0.0.1:41616 forcing change of non-runtime-safe flag fail_dns_resolution_hostports
I20260804 14:50:49.231482 25282 generic_service.cc:175] {username='slave'} at 127.0.0.1:41616 changed flags via RPC: fail_dns_resolution_hostports from '' to '127.24.6.130:39519'
W20260804 14:50:49.232731 25282 generic_service.cc:139] {username='slave'} at 127.0.0.1:41616 forcing change of non-runtime-safe flag fail_dns_resolution
I20260804 14:50:49.232805 25282 generic_service.cc:175] {username='slave'} at 127.0.0.1:41616 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
W20260804 14:50:49.238068 24924 generic_service.cc:139] {username='slave'} at 127.0.0.1:33906 forcing change of non-runtime-safe flag fail_dns_resolution_hostports
I20260804 14:50:49.238210 24924 generic_service.cc:175] {username='slave'} at 127.0.0.1:33906 changed flags via RPC: fail_dns_resolution_hostports from '' to '127.24.6.130:39519'
W20260804 14:50:49.239279 24924 generic_service.cc:139] {username='slave'} at 127.0.0.1:33906 forcing change of non-runtime-safe flag fail_dns_resolution
I20260804 14:50:49.239355 24924 generic_service.cc:175] {username='slave'} at 127.0.0.1:33906 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
I20260804 14:50:49.240762 25226 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c99d0efe445644f09815fc7c540f551d"
dest_uuid: "2f2a7b98753c479e9ce8d9db9d6490b0"
from {username='slave'} at 127.0.0.1:41604
I20260804 14:50:49.241204 25226 raft_consensus.cc:482] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260804 14:50:49.241394 25226 raft_consensus.cc:3037] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:50:49.242620 25226 raft_consensus.cc:504] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae609bd205d473797afce12a389988d" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39519 } } peers { permanent_uuid: "2f2a7b98753c479e9ce8d9db9d6490b0" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 35841 } } peers { permanent_uuid: "8d570a14f7d34a5d9affae0f3108d60c" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44825 } }
I20260804 14:50:49.244087 25226 leader_election.cc:288] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 [CANDIDATE]: Term 1 election: Requested vote from peers 4ae609bd205d473797afce12a389988d (127.24.6.130:39519), 8d570a14f7d34a5d9affae0f3108d60c (127.24.6.129:44825)
W20260804 14:50:49.244498 25325 leader_election.cc:334] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 4ae609bd205d473797afce12a389988d (127.24.6.130:39519): Network error: failed to refresh physical address: injected DNS resolution failure: 127.24.6.130:39519
I20260804 14:50:49.249004 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99d0efe445644f09815fc7c540f551d" candidate_uuid: "2f2a7b98753c479e9ce8d9db9d6490b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8d570a14f7d34a5d9affae0f3108d60c"
I20260804 14:50:49.249402 24867 raft_consensus.cc:3037] T c99d0efe445644f09815fc7c540f551d P 8d570a14f7d34a5d9affae0f3108d60c [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:50:49.250720 24867 raft_consensus.cc:2454] T c99d0efe445644f09815fc7c540f551d P 8d570a14f7d34a5d9affae0f3108d60c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f2a7b98753c479e9ce8d9db9d6490b0 in term 1.
I20260804 14:50:49.251288 25130 leader_election.cc:302] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f2a7b98753c479e9ce8d9db9d6490b0, 8d570a14f7d34a5d9affae0f3108d60c; no voters: 4ae609bd205d473797afce12a389988d
I20260804 14:50:49.251638 25321 raft_consensus.cc:2781] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:50:49.251953 25321 raft_consensus.cc:686] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 [term 1 LEADER]: Becoming Leader. State: Replica: 2f2a7b98753c479e9ce8d9db9d6490b0, State: Running, Role: LEADER
I20260804 14:50:49.252353 25321 consensus_queue.cc:237] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae609bd205d473797afce12a389988d" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39519 } } peers { permanent_uuid: "2f2a7b98753c479e9ce8d9db9d6490b0" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 35841 } } peers { permanent_uuid: "8d570a14f7d34a5d9affae0f3108d60c" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44825 } }
I20260804 14:50:49.256961 24658 catalog_manager.cc:5495] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f2a7b98753c479e9ce8d9db9d6490b0 (127.24.6.131). New cstate: current_term: 1 leader_uuid: "2f2a7b98753c479e9ce8d9db9d6490b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae609bd205d473797afce12a389988d" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f2a7b98753c479e9ce8d9db9d6490b0" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 35841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d570a14f7d34a5d9affae0f3108d60c" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44825 } health_report { overall_health: UNKNOWN } } }
I20260804 14:50:49.334218 25321 consensus_queue.cc:1035] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d570a14f7d34a5d9affae0f3108d60c" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260804 14:50:49.339852 25325 consensus_peers.cc:487] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 -> Peer 4ae609bd205d473797afce12a389988d (127.24.6.130:39519): Couldn't send request to peer 4ae609bd205d473797afce12a389988d. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.24.6.130:39519. This is attempt 1: this message will repeat every 5th retry.
W20260804 14:50:49.393319 25110 tablet.cc:2369] T c99d0efe445644f09815fc7c540f551d P 4ae609bd205d473797afce12a389988d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260804 14:50:49.819797 25325 consensus_peers.cc:487] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 -> Peer 4ae609bd205d473797afce12a389988d (127.24.6.130:39519): Couldn't send request to peer 4ae609bd205d473797afce12a389988d. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.24.6.130:39519. This is attempt 6: this message will repeat every 5th retry.
W20260804 14:50:50.312451 25325 consensus_peers.cc:487] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 -> Peer 4ae609bd205d473797afce12a389988d (127.24.6.130:39519): Couldn't send request to peer 4ae609bd205d473797afce12a389988d. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.24.6.130:39519. This is attempt 11: this message will repeat every 5th retry.
W20260804 14:50:50.842278 25325 consensus_peers.cc:487] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 -> Peer 4ae609bd205d473797afce12a389988d (127.24.6.130:39519): Couldn't send request to peer 4ae609bd205d473797afce12a389988d. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.24.6.130:39519. This is attempt 16: this message will repeat every 5th retry.
W20260804 14:50:51.360306 25325 consensus_peers.cc:487] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 -> Peer 4ae609bd205d473797afce12a389988d (127.24.6.130:39519): Couldn't send request to peer 4ae609bd205d473797afce12a389988d. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.24.6.130:39519. This is attempt 21: this message will repeat every 5th retry.
W20260804 14:50:51.858780 25325 consensus_peers.cc:487] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 -> Peer 4ae609bd205d473797afce12a389988d (127.24.6.130:39519): Couldn't send request to peer 4ae609bd205d473797afce12a389988d. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.24.6.130:39519. This is attempt 26: this message will repeat every 5th retry.
I20260804 14:50:52.326169 25331 consensus_queue.cc:579] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 [LEADER]: Leader has been unable to successfully communicate with peer 4ae609bd205d473797afce12a389988d for more than 3 seconds (3.073s)
W20260804 14:50:52.326902 25325 consensus_peers.cc:487] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 -> Peer 4ae609bd205d473797afce12a389988d (127.24.6.130:39519): Couldn't send request to peer 4ae609bd205d473797afce12a389988d. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.24.6.130:39519. This is attempt 31: this message will repeat every 5th retry.
W20260804 14:50:52.328712 25100 generic_service.cc:139] {username='slave'} at 127.0.0.1:57754 forcing change of non-runtime-safe flag fail_dns_resolution
W20260804 14:50:52.328737 24666 catalog_manager.cc:5173] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c99d0efe445644f09815fc7c540f551d with cas_config_opid_index -1: no extra replica candidate found for tablet c99d0efe445644f09815fc7c540f551d (table test-workload [id=15b6e4029344413792281b439ead2ccf]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20260804 14:50:52.328825 25100 generic_service.cc:175] {username='slave'} at 127.0.0.1:57754 changed flags via RPC: fail_dns_resolution from 'true' to 'false'
W20260804 14:50:52.329948 25282 generic_service.cc:139] {username='slave'} at 127.0.0.1:41616 forcing change of non-runtime-safe flag fail_dns_resolution
I20260804 14:50:52.330021 25282 generic_service.cc:175] {username='slave'} at 127.0.0.1:41616 changed flags via RPC: fail_dns_resolution from 'true' to 'false'
W20260804 14:50:52.331048 24924 generic_service.cc:139] {username='slave'} at 127.0.0.1:33906 forcing change of non-runtime-safe flag fail_dns_resolution
I20260804 14:50:52.331127 24924 generic_service.cc:175] {username='slave'} at 127.0.0.1:33906 changed flags via RPC: fail_dns_resolution from 'true' to 'false'
I20260804 14:50:52.409627 25044 raft_consensus.cc:3037] T c99d0efe445644f09815fc7c540f551d P 4ae609bd205d473797afce12a389988d [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:50:52.491827 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.132:0
--local_ip_for_outbound_sockets=127.24.6.132
--webserver_interface=127.24.6.132
--webserver_port=0
--tserver_master_addrs=127.24.6.190:39931
--builtin_ntp_servers=127.24.6.148:44795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--dns_resolver_cache_capacity_mb=0
--enable_leader_failure_detection=false
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=3
--leader_failure_max_missed_heartbeat_periods=10
--leader_failure_exp_backoff_max_delta_ms=5000 with env {}
W20260804 14:50:52.616849 25344 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260804 14:50:52.617271 25344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:50:52.617338 25344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:50:52.617425 25344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:50:52.622083 25344 flags.cc:425] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=5000
W20260804 14:50:52.622192 25344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:50:52.622278 25344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.132
I20260804 14:50:52.627782 25344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:44795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--enable_leader_failure_detection=false
--leader_failure_exp_backoff_max_delta_ms=5000
--leader_failure_max_missed_heartbeat_periods=10
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.24.6.132
--webserver_port=0
--tserver_master_addrs=127.24.6.190:39931
--never_fsync=true
--heap_profile_path=/tmp/kudu.25344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--dns_resolver_cache_capacity_mb=0
--local_ip_for_outbound_sockets=127.24.6.132
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:50:52.629163 25344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:50:52.630748 25344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:50:52.641132 25351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:50:52.641077 25350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:50:52.641115 25354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:50:52.641506 25344 server_base.cc:1034] running on GCE node
I20260804 14:50:52.642059 25344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:50:52.642849 25344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:50:52.644061 25344 hybrid_clock.cc:648] HybridClock initialized: now 1785855052644048 us; error 36 us; skew 500 ppm
I20260804 14:50:52.646222 25344 webserver.cc:460] Webserver started at http://127.24.6.132:33567/ using document root <none> and password file <none>
I20260804 14:50:52.646909 25344 fs_manager.cc:359] Metadata directory not provided
I20260804 14:50:52.647022 25344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:50:52.647329 25344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:50:52.649868 25344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-3/data/instance:
uuid: "01463dce07d44f02bb49c10b81786c52"
format_stamp: "Formatted at 2026-08-04 14:50:52 on dist-test-slave-92m1"
I20260804 14:50:52.650524 25344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-3/wal/instance:
uuid: "01463dce07d44f02bb49c10b81786c52"
format_stamp: "Formatted at 2026-08-04 14:50:52 on dist-test-slave-92m1"
I20260804 14:50:52.655550 25344 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260804 14:50:52.658202 25361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:50:52.659703 25344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260804 14:50:52.659850 25344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-3/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-3/wal
uuid: "01463dce07d44f02bb49c10b81786c52"
format_stamp: "Formatted at 2026-08-04 14:50:52 on dist-test-slave-92m1"
I20260804 14:50:52.659974 25344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:50:52.671805 25344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:50:52.672587 25344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:50:52.672823 25344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:50:52.673600 25344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:50:52.674664 25344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:50:52.674742 25344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:50:52.674820 25344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:50:52.674865 25344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:50:52.685420 25344 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.132:35081
I20260804 14:50:52.685437 25515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.132:35081 every 8 connection(s)
I20260804 14:50:52.686681 25344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-3/data/info.pb
I20260804 14:50:52.690430 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 25344
I20260804 14:50:52.690536 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0/raft_consensus_election-itest-cluster/ts-3/wal/instance
I20260804 14:50:52.698750 25517 heartbeater.cc:347] Connected to a master server at 127.24.6.190:39931
I20260804 14:50:52.699062 25517 heartbeater.cc:464] Registering TS with master...
I20260804 14:50:52.699726 25517 heartbeater.cc:511] Master 127.24.6.190:39931 requested a full tablet report, sending...
I20260804 14:50:52.700942 24666 ts_manager.cc:194] Registered new tserver with Master: 01463dce07d44f02bb49c10b81786c52 (127.24.6.132:35081)
I20260804 14:50:52.701735 24666 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.132:38705
I20260804 14:50:52.703020 25226 consensus_queue.cc:237] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ae609bd205d473797afce12a389988d" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39519 } } peers { permanent_uuid: "2f2a7b98753c479e9ce8d9db9d6490b0" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 35841 } } peers { permanent_uuid: "8d570a14f7d34a5d9affae0f3108d60c" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44825 } } peers { permanent_uuid: "01463dce07d44f02bb49c10b81786c52" member_type: NON_VOTER last_known_addr { host: "127.24.6.132" port: 35081 } attrs { promote: true } }
I20260804 14:50:52.705164 25044 raft_consensus.cc:1261] T c99d0efe445644f09815fc7c540f551d P 4ae609bd205d473797afce12a389988d [term 1 FOLLOWER]: Refusing update from remote peer 2f2a7b98753c479e9ce8d9db9d6490b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260804 14:50:52.705320 24867 raft_consensus.cc:1261] T c99d0efe445644f09815fc7c540f551d P 8d570a14f7d34a5d9affae0f3108d60c [term 1 FOLLOWER]: Refusing update from remote peer 2f2a7b98753c479e9ce8d9db9d6490b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260804 14:50:52.705808 25334 consensus_queue.cc:1035] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ae609bd205d473797afce12a389988d" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260804 14:50:52.706043 25331 consensus_queue.cc:1035] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d570a14f7d34a5d9affae0f3108d60c" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260804 14:50:52.708467 25333 raft_consensus.cc:2932] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01463dce07d44f02bb49c10b81786c52 (127.24.6.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ae609bd205d473797afce12a389988d" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39519 } } peers { permanent_uuid: "2f2a7b98753c479e9ce8d9db9d6490b0" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 35841 } } peers { permanent_uuid: "8d570a14f7d34a5d9affae0f3108d60c" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44825 } } peers { permanent_uuid: "01463dce07d44f02bb49c10b81786c52" member_type: NON_VOTER last_known_addr { host: "127.24.6.132" port: 35081 } attrs { promote: true } } }
I20260804 14:50:52.708899 25044 raft_consensus.cc:2932] T c99d0efe445644f09815fc7c540f551d P 4ae609bd205d473797afce12a389988d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01463dce07d44f02bb49c10b81786c52 (127.24.6.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ae609bd205d473797afce12a389988d" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39519 } } peers { permanent_uuid: "2f2a7b98753c479e9ce8d9db9d6490b0" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 35841 } } peers { permanent_uuid: "8d570a14f7d34a5d9affae0f3108d60c" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44825 } } peers { permanent_uuid: "01463dce07d44f02bb49c10b81786c52" member_type: NON_VOTER last_known_addr { host: "127.24.6.132" port: 35081 } attrs { promote: true } } }
I20260804 14:50:52.709311 24867 raft_consensus.cc:2932] T c99d0efe445644f09815fc7c540f551d P 8d570a14f7d34a5d9affae0f3108d60c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01463dce07d44f02bb49c10b81786c52 (127.24.6.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ae609bd205d473797afce12a389988d" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39519 } } peers { permanent_uuid: "2f2a7b98753c479e9ce8d9db9d6490b0" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 35841 } } peers { permanent_uuid: "8d570a14f7d34a5d9affae0f3108d60c" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44825 } } peers { permanent_uuid: "01463dce07d44f02bb49c10b81786c52" member_type: NON_VOTER last_known_addr { host: "127.24.6.132" port: 35081 } attrs { promote: true } } }
I20260804 14:50:52.711360 24640 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c99d0efe445644f09815fc7c540f551d with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 5)
I20260804 14:50:52.712308 24658 catalog_manager.cc:5495] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 reported cstate change: config changed from index -1 to 2, NON_VOTER 01463dce07d44f02bb49c10b81786c52 (127.24.6.132) added. New cstate: current_term: 1 leader_uuid: "2f2a7b98753c479e9ce8d9db9d6490b0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ae609bd205d473797afce12a389988d" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f2a7b98753c479e9ce8d9db9d6490b0" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 35841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d570a14f7d34a5d9affae0f3108d60c" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01463dce07d44f02bb49c10b81786c52" member_type: NON_VOTER last_known_addr { host: "127.24.6.132" port: 35081 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260804 14:50:52.713411 25133 consensus_peers.cc:487] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 -> Peer 01463dce07d44f02bb49c10b81786c52 (127.24.6.132:35081): Couldn't send request to peer 01463dce07d44f02bb49c10b81786c52. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c99d0efe445644f09815fc7c540f551d. This is attempt 1: this message will repeat every 5th retry.
I20260804 14:50:52.716091 25226 consensus_queue.cc:237] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ae609bd205d473797afce12a389988d" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39519 } } peers { permanent_uuid: "2f2a7b98753c479e9ce8d9db9d6490b0" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 35841 } } peers { permanent_uuid: "8d570a14f7d34a5d9affae0f3108d60c" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44825 } }
I20260804 14:50:52.717418 25044 raft_consensus.cc:1261] T c99d0efe445644f09815fc7c540f551d P 4ae609bd205d473797afce12a389988d [term 1 FOLLOWER]: Refusing update from remote peer 2f2a7b98753c479e9ce8d9db9d6490b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260804 14:50:52.717458 24867 raft_consensus.cc:1261] T c99d0efe445644f09815fc7c540f551d P 8d570a14f7d34a5d9affae0f3108d60c [term 1 FOLLOWER]: Refusing update from remote peer 2f2a7b98753c479e9ce8d9db9d6490b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260804 14:50:52.717844 25331 consensus_queue.cc:1035] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ae609bd205d473797afce12a389988d" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260804 14:50:52.717998 25334 consensus_queue.cc:1035] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d570a14f7d34a5d9affae0f3108d60c" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260804 14:50:52.719664 25334 raft_consensus.cc:2932] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 01463dce07d44f02bb49c10b81786c52 (127.24.6.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ae609bd205d473797afce12a389988d" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39519 } } peers { permanent_uuid: "2f2a7b98753c479e9ce8d9db9d6490b0" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 35841 } } peers { permanent_uuid: "8d570a14f7d34a5d9affae0f3108d60c" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44825 } } }
I20260804 14:50:52.720129 24867 raft_consensus.cc:2932] T c99d0efe445644f09815fc7c540f551d P 8d570a14f7d34a5d9affae0f3108d60c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 01463dce07d44f02bb49c10b81786c52 (127.24.6.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ae609bd205d473797afce12a389988d" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39519 } } peers { permanent_uuid: "2f2a7b98753c479e9ce8d9db9d6490b0" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 35841 } } peers { permanent_uuid: "8d570a14f7d34a5d9affae0f3108d60c" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44825 } } }
I20260804 14:50:52.720362 25044 raft_consensus.cc:2932] T c99d0efe445644f09815fc7c540f551d P 4ae609bd205d473797afce12a389988d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 01463dce07d44f02bb49c10b81786c52 (127.24.6.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ae609bd205d473797afce12a389988d" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39519 } } peers { permanent_uuid: "2f2a7b98753c479e9ce8d9db9d6490b0" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 35841 } } peers { permanent_uuid: "8d570a14f7d34a5d9affae0f3108d60c" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44825 } } }
I20260804 14:50:52.721995 24640 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet c99d0efe445644f09815fc7c540f551d with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260804 14:50:52.722798 24665 catalog_manager.cc:5495] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 reported cstate change: config changed from index 2 to 3, NON_VOTER 01463dce07d44f02bb49c10b81786c52 (127.24.6.132) evicted. New cstate: current_term: 1 leader_uuid: "2f2a7b98753c479e9ce8d9db9d6490b0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ae609bd205d473797afce12a389988d" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f2a7b98753c479e9ce8d9db9d6490b0" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 35841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d570a14f7d34a5d9affae0f3108d60c" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44825 } health_report { overall_health: HEALTHY } } }
I20260804 14:50:52.729959 25426 tablet_service.cc:1514] Processing DeleteTablet for tablet c99d0efe445644f09815fc7c540f551d with delete_type TABLET_DATA_TOMBSTONED (TS 01463dce07d44f02bb49c10b81786c52 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:35228
W20260804 14:50:52.731511 24642 catalog_manager.cc:4808] TS 01463dce07d44f02bb49c10b81786c52 (127.24.6.132:35081): delete failed for tablet c99d0efe445644f09815fc7c540f551d because the tablet was not found. No further retry: Not found: Tablet not found: c99d0efe445644f09815fc7c540f551d
W20260804 14:50:52.744683 25100 generic_service.cc:139] {username='slave'} at 127.0.0.1:57754 forcing change of non-runtime-safe flag fail_dns_resolution
I20260804 14:50:52.744800 25100 generic_service.cc:175] {username='slave'} at 127.0.0.1:57754 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
W20260804 14:50:52.745924 25282 generic_service.cc:139] {username='slave'} at 127.0.0.1:41616 forcing change of non-runtime-safe flag fail_dns_resolution
I20260804 14:50:52.746039 25282 generic_service.cc:175] {username='slave'} at 127.0.0.1:41616 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
W20260804 14:50:52.747112 24924 generic_service.cc:139] {username='slave'} at 127.0.0.1:33906 forcing change of non-runtime-safe flag fail_dns_resolution
I20260804 14:50:52.747251 24924 generic_service.cc:175] {username='slave'} at 127.0.0.1:33906 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
I20260804 14:50:52.748308 25226 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c99d0efe445644f09815fc7c540f551d"
dest_uuid: "2f2a7b98753c479e9ce8d9db9d6490b0"
mode: GRACEFUL
new_leader_uuid: "2f2a7b98753c479e9ce8d9db9d6490b0"
from {username='slave'} at 127.0.0.1:41604
I20260804 14:50:52.748567 25226 raft_consensus.cc:595] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 [term 1 LEADER]: Received request to transfer leadership to TS 2f2a7b98753c479e9ce8d9db9d6490b0
I20260804 14:50:53.704259 25517 heartbeater.cc:503] Master 127.24.6.190:39931 was elected leader, sending a full tablet report...
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/raft_consensus_election-itest.cc:292: Failure
Value of: tablets.empty()
Actual: true
Expected: false
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_util.cc:395: Failure
Failed
Timed out waiting for assertion to pass.
I20260804 14:51:22.841002 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 24750
W20260804 14:51:22.857357 25130 connection.cc:535] client connection to 127.24.6.129:44825 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260804 14:51:22.857748 25130 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260804 14:51:22.857822 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 24934
W20260804 14:51:22.866752 25130 consensus_peers.cc:487] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 -> Peer 8d570a14f7d34a5d9affae0f3108d60c (127.24.6.129:44825): Couldn't send request to peer 8d570a14f7d34a5d9affae0f3108d60c. Status: Network error: Client connection negotiation failed: client connection to 127.24.6.129:44825: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260804 14:51:22.872627 25136 connection.cc:535] client connection to 127.24.6.130:39519 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260804 14:51:22.873234 25551 consensus_peers.cc:487] T c99d0efe445644f09815fc7c540f551d P 2f2a7b98753c479e9ce8d9db9d6490b0 -> Peer 4ae609bd205d473797afce12a389988d (127.24.6.130:39519): Couldn't send request to peer 4ae609bd205d473797afce12a389988d. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.24.6.130:39519. This is attempt 1: this message will repeat every 5th retry.
I20260804 14:51:22.873276 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 25112
I20260804 14:51:22.885605 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 25344
I20260804 14:51:22.899858 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 24617
2026-08-04T14:51:22Z chronyd exiting
I20260804 14:51:22.924295 24602 test_util.cc:175] -----------------------------------------------
I20260804 14:51:22.924362 24602 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1785855048081067-24602-0
[ FAILED ] RaftConsensusElectionITest.TestNewLeaderCantResolvePeers (34788 ms)
[ RUN ] RaftConsensusElectionITest.RunLeaderElection
I20260804 14:51:22.925160 24602 test_util.cc:268] Using random seed: -908686235
I20260804 14:51:22.926018 24602 ts_itest-base.cc:112] Starting cluster with:
I20260804 14:51:22.926056 24602 ts_itest-base.cc:113] --------------
I20260804 14:51:22.926069 24602 ts_itest-base.cc:114] 3 tablet servers
I20260804 14:51:22.926079 24602 ts_itest-base.cc:115] 3 replicas per TS
I20260804 14:51:22.926090 24602 ts_itest-base.cc:116] --------------
2026-08-04T14:51:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-04T14:51:22Z Disabled control of system clock
I20260804 14:51:22.937490 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.6.190:39907
--webserver_interface=127.24.6.190
--webserver_port=0
--builtin_ntp_servers=127.24.6.148:38921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.6.190:39907 with env {}
W20260804 14:51:23.056501 25562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:51:23.056856 25562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:51:23.056924 25562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:51:23.061061 25562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260804 14:51:23.061162 25562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:51:23.061198 25562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260804 14:51:23.061235 25562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260804 14:51:23.066038 25562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:38921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.6.190:39907
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.6.190:39907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.24.6.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.25562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:51:23.067289 25562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:51:23.068770 25562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260804 14:51:23.078071 25562 server_base.cc:1034] running on GCE node
W20260804 14:51:23.078150 25567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:51:23.078284 25572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:51:23.078162 25568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:51:23.078944 25562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:51:23.080291 25562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:51:23.081509 25562 hybrid_clock.cc:648] HybridClock initialized: now 1785855083081484 us; error 38 us; skew 500 ppm
I20260804 14:51:23.087858 25562 webserver.cc:460] Webserver started at http://127.24.6.190:46195/ using document root <none> and password file <none>
I20260804 14:51:23.088550 25562 fs_manager.cc:359] Metadata directory not provided
I20260804 14:51:23.088655 25562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:51:23.088925 25562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:51:23.091099 25562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d2f5298df06d41c6b4954af3bc6f6984"
format_stamp: "Formatted at 2026-08-04 14:51:23 on dist-test-slave-92m1"
I20260804 14:51:23.091766 25562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d2f5298df06d41c6b4954af3bc6f6984"
format_stamp: "Formatted at 2026-08-04 14:51:23 on dist-test-slave-92m1"
I20260804 14:51:23.096133 25562 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260804 14:51:23.098723 25582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:51:23.100131 25562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260804 14:51:23.100293 25562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d2f5298df06d41c6b4954af3bc6f6984"
format_stamp: "Formatted at 2026-08-04 14:51:23 on dist-test-slave-92m1"
I20260804 14:51:23.100418 25562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:51:23.117743 25562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:51:23.118506 25562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:51:23.118733 25562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:51:23.126762 25562 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.190:39907
I20260804 14:51:23.126778 25663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.190:39907 every 8 connection(s)
I20260804 14:51:23.127867 25562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260804 14:51:23.131742 25664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:51:23.134392 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 25562
I20260804 14:51:23.134495 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260804 14:51:23.137389 25664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f5298df06d41c6b4954af3bc6f6984: Bootstrap starting.
I20260804 14:51:23.140072 25664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2f5298df06d41c6b4954af3bc6f6984: Neither blocks nor log segments found. Creating new log.
I20260804 14:51:23.141049 25664 log.cc:824] T 00000000000000000000000000000000 P d2f5298df06d41c6b4954af3bc6f6984: Log is configured to *not* fsync() on all Append() calls
I20260804 14:51:23.143478 25664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f5298df06d41c6b4954af3bc6f6984: No bootstrap required, opened a new log
I20260804 14:51:23.147477 25664 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2f5298df06d41c6b4954af3bc6f6984 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f5298df06d41c6b4954af3bc6f6984" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 39907 } }
I20260804 14:51:23.147730 25664 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2f5298df06d41c6b4954af3bc6f6984 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:51:23.147768 25664 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2f5298df06d41c6b4954af3bc6f6984 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2f5298df06d41c6b4954af3bc6f6984, State: Initialized, Role: FOLLOWER
I20260804 14:51:23.148226 25664 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2f5298df06d41c6b4954af3bc6f6984 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f5298df06d41c6b4954af3bc6f6984" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 39907 } }
I20260804 14:51:23.148345 25664 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2f5298df06d41c6b4954af3bc6f6984 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260804 14:51:23.148413 25664 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2f5298df06d41c6b4954af3bc6f6984 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260804 14:51:23.148514 25664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2f5298df06d41c6b4954af3bc6f6984 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:51:23.149458 25664 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2f5298df06d41c6b4954af3bc6f6984 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f5298df06d41c6b4954af3bc6f6984" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 39907 } }
I20260804 14:51:23.149823 25664 leader_election.cc:302] T 00000000000000000000000000000000 P d2f5298df06d41c6b4954af3bc6f6984 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2f5298df06d41c6b4954af3bc6f6984; no voters:
I20260804 14:51:23.150182 25664 leader_election.cc:288] T 00000000000000000000000000000000 P d2f5298df06d41c6b4954af3bc6f6984 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260804 14:51:23.150308 25669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2f5298df06d41c6b4954af3bc6f6984 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:51:23.150588 25669 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2f5298df06d41c6b4954af3bc6f6984 [term 1 LEADER]: Becoming Leader. State: Replica: d2f5298df06d41c6b4954af3bc6f6984, State: Running, Role: LEADER
I20260804 14:51:23.150983 25669 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2f5298df06d41c6b4954af3bc6f6984 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f5298df06d41c6b4954af3bc6f6984" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 39907 } }
I20260804 14:51:23.151613 25664 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2f5298df06d41c6b4954af3bc6f6984 [sys.catalog]: configured and running, proceeding with master startup.
I20260804 14:51:23.153049 25670 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f5298df06d41c6b4954af3bc6f6984 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2f5298df06d41c6b4954af3bc6f6984" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f5298df06d41c6b4954af3bc6f6984" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 39907 } } }
I20260804 14:51:23.153193 25670 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f5298df06d41c6b4954af3bc6f6984 [sys.catalog]: This master's current role is: LEADER
I20260804 14:51:23.153510 25672 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f5298df06d41c6b4954af3bc6f6984 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2f5298df06d41c6b4954af3bc6f6984. Latest consensus state: current_term: 1 leader_uuid: "d2f5298df06d41c6b4954af3bc6f6984" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f5298df06d41c6b4954af3bc6f6984" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 39907 } } }
I20260804 14:51:23.153613 25672 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f5298df06d41c6b4954af3bc6f6984 [sys.catalog]: This master's current role is: LEADER
I20260804 14:51:23.153668 25680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260804 14:51:23.157316 25680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260804 14:51:23.164093 25680 catalog_manager.cc:1347] Generated new cluster ID: 4b58f76ab21a4d57961719afed6596dc
I20260804 14:51:23.164184 25680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260804 14:51:23.173957 25680 catalog_manager.cc:1370] Generated new certificate authority record
I20260804 14:51:23.175405 25680 catalog_manager.cc:1504] Loading token signing keys...
I20260804 14:51:23.187845 25680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2f5298df06d41c6b4954af3bc6f6984: Generated new TSK 0
I20260804 14:51:23.188850 25680 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260804 14:51:23.201390 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.129:0
--local_ip_for_outbound_sockets=127.24.6.129
--webserver_interface=127.24.6.129
--webserver_port=0
--tserver_master_addrs=127.24.6.190:39907
--builtin_ntp_servers=127.24.6.148:38921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
W20260804 14:51:23.328837 25695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:51:23.329185 25695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:51:23.329267 25695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:51:23.333621 25695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:51:23.333739 25695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.129
I20260804 14:51:23.338900 25695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:38921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.24.6.129
--webserver_port=0
--tserver_master_addrs=127.24.6.190:39907
--never_fsync=true
--heap_profile_path=/tmp/kudu.25695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.129
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:51:23.340240 25695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:51:23.341775 25695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:51:23.352401 25701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:51:23.352567 25695 server_base.cc:1034] running on GCE node
W20260804 14:51:23.352406 25703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:51:23.352799 25700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:51:23.353313 25695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:51:23.354085 25695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:51:23.355331 25695 hybrid_clock.cc:648] HybridClock initialized: now 1785855083355303 us; error 36 us; skew 500 ppm
I20260804 14:51:23.357610 25695 webserver.cc:460] Webserver started at http://127.24.6.129:42527/ using document root <none> and password file <none>
I20260804 14:51:23.358304 25695 fs_manager.cc:359] Metadata directory not provided
I20260804 14:51:23.358409 25695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:51:23.358664 25695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:51:23.360965 25695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "000ec79ca0274f6b83147bfc56d600ad"
format_stamp: "Formatted at 2026-08-04 14:51:23 on dist-test-slave-92m1"
I20260804 14:51:23.361708 25695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "000ec79ca0274f6b83147bfc56d600ad"
format_stamp: "Formatted at 2026-08-04 14:51:23 on dist-test-slave-92m1"
I20260804 14:51:23.366003 25695 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260804 14:51:23.368938 25712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:51:23.370492 25695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260804 14:51:23.370641 25695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "000ec79ca0274f6b83147bfc56d600ad"
format_stamp: "Formatted at 2026-08-04 14:51:23 on dist-test-slave-92m1"
I20260804 14:51:23.370770 25695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:51:23.386168 25695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:51:23.387032 25695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:51:23.387326 25695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:51:23.388032 25695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:51:23.389130 25695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:51:23.389190 25695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:51:23.389236 25695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:51:23.389294 25695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:51:23.399519 25695 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.129:34809
I20260804 14:51:23.399542 25862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.129:34809 every 8 connection(s)
I20260804 14:51:23.400691 25695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260804 14:51:23.410130 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 25695
I20260804 14:51:23.410288 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260804 14:51:23.413070 25864 heartbeater.cc:347] Connected to a master server at 127.24.6.190:39907
I20260804 14:51:23.413143 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.130:0
--local_ip_for_outbound_sockets=127.24.6.130
--webserver_interface=127.24.6.130
--webserver_port=0
--tserver_master_addrs=127.24.6.190:39907
--builtin_ntp_servers=127.24.6.148:38921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
I20260804 14:51:23.413376 25864 heartbeater.cc:464] Registering TS with master...
I20260804 14:51:23.414001 25864 heartbeater.cc:511] Master 127.24.6.190:39907 requested a full tablet report, sending...
I20260804 14:51:23.416013 25611 ts_manager.cc:194] Registered new tserver with Master: 000ec79ca0274f6b83147bfc56d600ad (127.24.6.129:34809)
I20260804 14:51:23.417375 25611 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.129:46979
W20260804 14:51:23.537652 25867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:51:23.537957 25867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:51:23.537994 25867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:51:23.544426 25867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:51:23.544605 25867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.130
I20260804 14:51:23.551772 25867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:38921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.24.6.130
--webserver_port=0
--tserver_master_addrs=127.24.6.190:39907
--never_fsync=true
--heap_profile_path=/tmp/kudu.25867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.130
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:51:23.553112 25867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:51:23.554600 25867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:51:23.565011 25875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:51:23.565034 25873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:51:23.565241 25872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:51:23.565428 25867 server_base.cc:1034] running on GCE node
I20260804 14:51:23.566007 25867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:51:23.566715 25867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:51:23.567899 25867 hybrid_clock.cc:648] HybridClock initialized: now 1785855083567857 us; error 54 us; skew 500 ppm
I20260804 14:51:23.569958 25867 webserver.cc:460] Webserver started at http://127.24.6.130:33335/ using document root <none> and password file <none>
I20260804 14:51:23.570608 25867 fs_manager.cc:359] Metadata directory not provided
I20260804 14:51:23.570706 25867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:51:23.570986 25867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:51:23.573218 25867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "8fcb2230014a476a8eb1eff1f6625fd2"
format_stamp: "Formatted at 2026-08-04 14:51:23 on dist-test-slave-92m1"
I20260804 14:51:23.573920 25867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "8fcb2230014a476a8eb1eff1f6625fd2"
format_stamp: "Formatted at 2026-08-04 14:51:23 on dist-test-slave-92m1"
I20260804 14:51:23.578305 25867 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260804 14:51:23.581151 25882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:51:23.582531 25867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260804 14:51:23.582674 25867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "8fcb2230014a476a8eb1eff1f6625fd2"
format_stamp: "Formatted at 2026-08-04 14:51:23 on dist-test-slave-92m1"
I20260804 14:51:23.582794 25867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:51:23.599406 25867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:51:23.600231 25867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:51:23.600455 25867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:51:23.601115 25867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:51:23.602206 25867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:51:23.602279 25867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:51:23.602351 25867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:51:23.602398 25867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:51:23.613014 25867 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.130:43911
I20260804 14:51:23.613042 26036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.130:43911 every 8 connection(s)
I20260804 14:51:23.614040 25867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260804 14:51:23.623351 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 25867
I20260804 14:51:23.623498 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260804 14:51:23.626135 26038 heartbeater.cc:347] Connected to a master server at 127.24.6.190:39907
I20260804 14:51:23.626448 26038 heartbeater.cc:464] Registering TS with master...
I20260804 14:51:23.626618 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.131:0
--local_ip_for_outbound_sockets=127.24.6.131
--webserver_interface=127.24.6.131
--webserver_port=0
--tserver_master_addrs=127.24.6.190:39907
--builtin_ntp_servers=127.24.6.148:38921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
I20260804 14:51:23.627130 26038 heartbeater.cc:511] Master 127.24.6.190:39907 requested a full tablet report, sending...
I20260804 14:51:23.628427 25610 ts_manager.cc:194] Registered new tserver with Master: 8fcb2230014a476a8eb1eff1f6625fd2 (127.24.6.130:43911)
I20260804 14:51:23.629087 25610 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.130:52657
W20260804 14:51:23.753306 26041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:51:23.753619 26041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:51:23.753695 26041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:51:23.758162 26041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:51:23.758271 26041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.131
I20260804 14:51:23.763443 26041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:38921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.24.6.131
--webserver_port=0
--tserver_master_addrs=127.24.6.190:39907
--never_fsync=true
--heap_profile_path=/tmp/kudu.26041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.131
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:51:23.764657 26041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:51:23.766211 26041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:51:23.776834 26050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:51:23.776911 26041 server_base.cc:1034] running on GCE node
W20260804 14:51:23.776844 26048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:51:23.776860 26047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:51:23.777585 26041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:51:23.778301 26041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:51:23.779510 26041 hybrid_clock.cc:648] HybridClock initialized: now 1785855083779492 us; error 33 us; skew 500 ppm
I20260804 14:51:23.781658 26041 webserver.cc:460] Webserver started at http://127.24.6.131:46235/ using document root <none> and password file <none>
I20260804 14:51:23.782326 26041 fs_manager.cc:359] Metadata directory not provided
I20260804 14:51:23.782394 26041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:51:23.782639 26041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:51:23.784901 26041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "cdc5e32317de46d2ab6214138fd191e6"
format_stamp: "Formatted at 2026-08-04 14:51:23 on dist-test-slave-92m1"
I20260804 14:51:23.785526 26041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "cdc5e32317de46d2ab6214138fd191e6"
format_stamp: "Formatted at 2026-08-04 14:51:23 on dist-test-slave-92m1"
I20260804 14:51:23.789896 26041 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260804 14:51:23.792891 26060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:51:23.794445 26041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260804 14:51:23.794596 26041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "cdc5e32317de46d2ab6214138fd191e6"
format_stamp: "Formatted at 2026-08-04 14:51:23 on dist-test-slave-92m1"
I20260804 14:51:23.794723 26041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:51:23.809113 26041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:51:23.809928 26041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:51:23.810173 26041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:51:23.810853 26041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:51:23.812008 26041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:51:23.812069 26041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:51:23.812139 26041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:51:23.812172 26041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:51:23.821882 26041 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.131:41355
I20260804 14:51:23.821907 26227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.131:41355 every 8 connection(s)
I20260804 14:51:23.822952 26041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260804 14:51:23.825606 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 26041
I20260804 14:51:23.825724 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260804 14:51:23.835143 26228 heartbeater.cc:347] Connected to a master server at 127.24.6.190:39907
I20260804 14:51:23.835484 26228 heartbeater.cc:464] Registering TS with master...
I20260804 14:51:23.836092 26228 heartbeater.cc:511] Master 127.24.6.190:39907 requested a full tablet report, sending...
I20260804 14:51:23.837255 25610 ts_manager.cc:194] Registered new tserver with Master: cdc5e32317de46d2ab6214138fd191e6 (127.24.6.131:41355)
I20260804 14:51:23.837832 25610 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.131:34959
I20260804 14:51:23.840404 24602 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260804 14:51:23.851015 25610 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40888:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20260804 14:51:23.854698 25610 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260804 14:51:23.874826 26132 tablet_service.cc:1467] Processing CreateTablet for tablet 405aa67ad6d14cc5bacd40d24aa238d6 (DEFAULT_TABLE table=TestTable [id=6b2251048cb64307b074835b1fd80dfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:51:23.874889 25954 tablet_service.cc:1467] Processing CreateTablet for tablet 405aa67ad6d14cc5bacd40d24aa238d6 (DEFAULT_TABLE table=TestTable [id=6b2251048cb64307b074835b1fd80dfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:51:23.876088 25954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 405aa67ad6d14cc5bacd40d24aa238d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:51:23.876102 26132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 405aa67ad6d14cc5bacd40d24aa238d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:51:23.876034 25771 tablet_service.cc:1467] Processing CreateTablet for tablet 405aa67ad6d14cc5bacd40d24aa238d6 (DEFAULT_TABLE table=TestTable [id=6b2251048cb64307b074835b1fd80dfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:51:23.877060 25771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 405aa67ad6d14cc5bacd40d24aa238d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:51:23.882337 26255 tablet_bootstrap.cc:492] T 405aa67ad6d14cc5bacd40d24aa238d6 P cdc5e32317de46d2ab6214138fd191e6: Bootstrap starting.
I20260804 14:51:23.883760 26256 tablet_bootstrap.cc:492] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2: Bootstrap starting.
I20260804 14:51:23.884466 26261 tablet_bootstrap.cc:492] T 405aa67ad6d14cc5bacd40d24aa238d6 P 000ec79ca0274f6b83147bfc56d600ad: Bootstrap starting.
I20260804 14:51:23.885340 26255 tablet_bootstrap.cc:654] T 405aa67ad6d14cc5bacd40d24aa238d6 P cdc5e32317de46d2ab6214138fd191e6: Neither blocks nor log segments found. Creating new log.
I20260804 14:51:23.886189 26256 tablet_bootstrap.cc:654] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2: Neither blocks nor log segments found. Creating new log.
I20260804 14:51:23.886579 26261 tablet_bootstrap.cc:654] T 405aa67ad6d14cc5bacd40d24aa238d6 P 000ec79ca0274f6b83147bfc56d600ad: Neither blocks nor log segments found. Creating new log.
I20260804 14:51:23.886914 26255 log.cc:824] T 405aa67ad6d14cc5bacd40d24aa238d6 P cdc5e32317de46d2ab6214138fd191e6: Log is configured to *not* fsync() on all Append() calls
I20260804 14:51:23.887035 26256 log.cc:824] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2: Log is configured to *not* fsync() on all Append() calls
I20260804 14:51:23.887445 26261 log.cc:824] T 405aa67ad6d14cc5bacd40d24aa238d6 P 000ec79ca0274f6b83147bfc56d600ad: Log is configured to *not* fsync() on all Append() calls
I20260804 14:51:23.889153 26256 tablet_bootstrap.cc:492] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2: No bootstrap required, opened a new log
I20260804 14:51:23.889191 26255 tablet_bootstrap.cc:492] T 405aa67ad6d14cc5bacd40d24aa238d6 P cdc5e32317de46d2ab6214138fd191e6: No bootstrap required, opened a new log
I20260804 14:51:23.889199 26261 tablet_bootstrap.cc:492] T 405aa67ad6d14cc5bacd40d24aa238d6 P 000ec79ca0274f6b83147bfc56d600ad: No bootstrap required, opened a new log
I20260804 14:51:23.889395 26261 ts_tablet_manager.cc:1397] T 405aa67ad6d14cc5bacd40d24aa238d6 P 000ec79ca0274f6b83147bfc56d600ad: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260804 14:51:23.889395 26255 ts_tablet_manager.cc:1397] T 405aa67ad6d14cc5bacd40d24aa238d6 P cdc5e32317de46d2ab6214138fd191e6: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260804 14:51:23.889401 26256 ts_tablet_manager.cc:1397] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260804 14:51:23.893301 26256 raft_consensus.cc:348] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcb2230014a476a8eb1eff1f6625fd2" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43911 } } peers { permanent_uuid: "cdc5e32317de46d2ab6214138fd191e6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41355 } } peers { permanent_uuid: "000ec79ca0274f6b83147bfc56d600ad" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34809 } }
I20260804 14:51:23.893695 26256 raft_consensus.cc:374] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:51:23.893813 26256 raft_consensus.cc:729] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fcb2230014a476a8eb1eff1f6625fd2, State: Initialized, Role: FOLLOWER
I20260804 14:51:23.894222 26261 raft_consensus.cc:348] T 405aa67ad6d14cc5bacd40d24aa238d6 P 000ec79ca0274f6b83147bfc56d600ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcb2230014a476a8eb1eff1f6625fd2" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43911 } } peers { permanent_uuid: "cdc5e32317de46d2ab6214138fd191e6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41355 } } peers { permanent_uuid: "000ec79ca0274f6b83147bfc56d600ad" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34809 } }
I20260804 14:51:23.894212 26255 raft_consensus.cc:348] T 405aa67ad6d14cc5bacd40d24aa238d6 P cdc5e32317de46d2ab6214138fd191e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcb2230014a476a8eb1eff1f6625fd2" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43911 } } peers { permanent_uuid: "cdc5e32317de46d2ab6214138fd191e6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41355 } } peers { permanent_uuid: "000ec79ca0274f6b83147bfc56d600ad" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34809 } }
I20260804 14:51:23.894482 26255 raft_consensus.cc:374] T 405aa67ad6d14cc5bacd40d24aa238d6 P cdc5e32317de46d2ab6214138fd191e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:51:23.894403 26256 consensus_queue.cc:260] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcb2230014a476a8eb1eff1f6625fd2" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43911 } } peers { permanent_uuid: "cdc5e32317de46d2ab6214138fd191e6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41355 } } peers { permanent_uuid: "000ec79ca0274f6b83147bfc56d600ad" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34809 } }
I20260804 14:51:23.894556 26255 raft_consensus.cc:729] T 405aa67ad6d14cc5bacd40d24aa238d6 P cdc5e32317de46d2ab6214138fd191e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdc5e32317de46d2ab6214138fd191e6, State: Initialized, Role: FOLLOWER
I20260804 14:51:23.894666 26261 raft_consensus.cc:374] T 405aa67ad6d14cc5bacd40d24aa238d6 P 000ec79ca0274f6b83147bfc56d600ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:51:23.894778 26261 raft_consensus.cc:729] T 405aa67ad6d14cc5bacd40d24aa238d6 P 000ec79ca0274f6b83147bfc56d600ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 000ec79ca0274f6b83147bfc56d600ad, State: Initialized, Role: FOLLOWER
I20260804 14:51:23.895023 26255 consensus_queue.cc:260] T 405aa67ad6d14cc5bacd40d24aa238d6 P cdc5e32317de46d2ab6214138fd191e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcb2230014a476a8eb1eff1f6625fd2" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43911 } } peers { permanent_uuid: "cdc5e32317de46d2ab6214138fd191e6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41355 } } peers { permanent_uuid: "000ec79ca0274f6b83147bfc56d600ad" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34809 } }
I20260804 14:51:23.895371 26038 heartbeater.cc:503] Master 127.24.6.190:39907 was elected leader, sending a full tablet report...
I20260804 14:51:23.895295 26261 consensus_queue.cc:260] T 405aa67ad6d14cc5bacd40d24aa238d6 P 000ec79ca0274f6b83147bfc56d600ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcb2230014a476a8eb1eff1f6625fd2" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43911 } } peers { permanent_uuid: "cdc5e32317de46d2ab6214138fd191e6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41355 } } peers { permanent_uuid: "000ec79ca0274f6b83147bfc56d600ad" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34809 } }
I20260804 14:51:23.895787 26256 ts_tablet_manager.cc:1428] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260804 14:51:23.896132 26255 ts_tablet_manager.cc:1428] T 405aa67ad6d14cc5bacd40d24aa238d6 P cdc5e32317de46d2ab6214138fd191e6: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260804 14:51:23.896165 26261 ts_tablet_manager.cc:1428] T 405aa67ad6d14cc5bacd40d24aa238d6 P 000ec79ca0274f6b83147bfc56d600ad: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260804 14:51:23.896524 26228 heartbeater.cc:503] Master 127.24.6.190:39907 was elected leader, sending a full tablet report...
I20260804 14:51:23.896878 25864 heartbeater.cc:503] Master 127.24.6.190:39907 was elected leader, sending a full tablet report...
W20260804 14:51:23.901988 25865 tablet.cc:2369] T 405aa67ad6d14cc5bacd40d24aa238d6 P 000ec79ca0274f6b83147bfc56d600ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260804 14:51:24.074318 26230 tablet.cc:2369] T 405aa67ad6d14cc5bacd40d24aa238d6 P cdc5e32317de46d2ab6214138fd191e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260804 14:51:24.115386 26039 tablet.cc:2369] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260804 14:51:24.255337 26266 raft_consensus.cc:482] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260804 14:51:24.255646 26266 raft_consensus.cc:504] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcb2230014a476a8eb1eff1f6625fd2" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43911 } } peers { permanent_uuid: "cdc5e32317de46d2ab6214138fd191e6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41355 } } peers { permanent_uuid: "000ec79ca0274f6b83147bfc56d600ad" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34809 } }
I20260804 14:51:24.256955 26266 leader_election.cc:288] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdc5e32317de46d2ab6214138fd191e6 (127.24.6.131:41355), 000ec79ca0274f6b83147bfc56d600ad (127.24.6.129:34809)
I20260804 14:51:24.261864 26162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "405aa67ad6d14cc5bacd40d24aa238d6" candidate_uuid: "8fcb2230014a476a8eb1eff1f6625fd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdc5e32317de46d2ab6214138fd191e6" is_pre_election: true
I20260804 14:51:24.261865 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "405aa67ad6d14cc5bacd40d24aa238d6" candidate_uuid: "8fcb2230014a476a8eb1eff1f6625fd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "000ec79ca0274f6b83147bfc56d600ad" is_pre_election: true
I20260804 14:51:24.262230 26162 raft_consensus.cc:2454] T 405aa67ad6d14cc5bacd40d24aa238d6 P cdc5e32317de46d2ab6214138fd191e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fcb2230014a476a8eb1eff1f6625fd2 in term 0.
I20260804 14:51:24.262230 25800 raft_consensus.cc:2454] T 405aa67ad6d14cc5bacd40d24aa238d6 P 000ec79ca0274f6b83147bfc56d600ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fcb2230014a476a8eb1eff1f6625fd2 in term 0.
I20260804 14:51:24.262773 25889 leader_election.cc:302] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8fcb2230014a476a8eb1eff1f6625fd2, cdc5e32317de46d2ab6214138fd191e6; no voters:
I20260804 14:51:24.263049 26266 raft_consensus.cc:2781] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260804 14:51:24.263149 26266 raft_consensus.cc:482] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260804 14:51:24.263221 26266 raft_consensus.cc:3037] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:51:24.264238 26266 raft_consensus.cc:504] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcb2230014a476a8eb1eff1f6625fd2" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43911 } } peers { permanent_uuid: "cdc5e32317de46d2ab6214138fd191e6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41355 } } peers { permanent_uuid: "000ec79ca0274f6b83147bfc56d600ad" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34809 } }
I20260804 14:51:24.264617 26266 leader_election.cc:288] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2 [CANDIDATE]: Term 1 election: Requested vote from peers cdc5e32317de46d2ab6214138fd191e6 (127.24.6.131:41355), 000ec79ca0274f6b83147bfc56d600ad (127.24.6.129:34809)
I20260804 14:51:24.265007 26162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "405aa67ad6d14cc5bacd40d24aa238d6" candidate_uuid: "8fcb2230014a476a8eb1eff1f6625fd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdc5e32317de46d2ab6214138fd191e6"
I20260804 14:51:24.265017 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "405aa67ad6d14cc5bacd40d24aa238d6" candidate_uuid: "8fcb2230014a476a8eb1eff1f6625fd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "000ec79ca0274f6b83147bfc56d600ad"
I20260804 14:51:24.265174 26162 raft_consensus.cc:3037] T 405aa67ad6d14cc5bacd40d24aa238d6 P cdc5e32317de46d2ab6214138fd191e6 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:51:24.265173 25800 raft_consensus.cc:3037] T 405aa67ad6d14cc5bacd40d24aa238d6 P 000ec79ca0274f6b83147bfc56d600ad [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:51:24.266206 25800 raft_consensus.cc:2454] T 405aa67ad6d14cc5bacd40d24aa238d6 P 000ec79ca0274f6b83147bfc56d600ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fcb2230014a476a8eb1eff1f6625fd2 in term 1.
I20260804 14:51:24.266323 26162 raft_consensus.cc:2454] T 405aa67ad6d14cc5bacd40d24aa238d6 P cdc5e32317de46d2ab6214138fd191e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fcb2230014a476a8eb1eff1f6625fd2 in term 1.
I20260804 14:51:24.266538 25890 leader_election.cc:302] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 000ec79ca0274f6b83147bfc56d600ad, 8fcb2230014a476a8eb1eff1f6625fd2; no voters:
I20260804 14:51:24.266752 26266 raft_consensus.cc:2781] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:51:24.267027 26266 raft_consensus.cc:686] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2 [term 1 LEADER]: Becoming Leader. State: Replica: 8fcb2230014a476a8eb1eff1f6625fd2, State: Running, Role: LEADER
I20260804 14:51:24.267446 26266 consensus_queue.cc:237] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcb2230014a476a8eb1eff1f6625fd2" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43911 } } peers { permanent_uuid: "cdc5e32317de46d2ab6214138fd191e6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41355 } } peers { permanent_uuid: "000ec79ca0274f6b83147bfc56d600ad" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34809 } }
I20260804 14:51:24.270937 25611 catalog_manager.cc:5495] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8fcb2230014a476a8eb1eff1f6625fd2 (127.24.6.130). New cstate: current_term: 1 leader_uuid: "8fcb2230014a476a8eb1eff1f6625fd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcb2230014a476a8eb1eff1f6625fd2" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdc5e32317de46d2ab6214138fd191e6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "000ec79ca0274f6b83147bfc56d600ad" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34809 } health_report { overall_health: UNKNOWN } } }
I20260804 14:51:24.322997 24602 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260804 14:51:24.325881 24602 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 000ec79ca0274f6b83147bfc56d600ad to finish bootstrapping
I20260804 14:51:24.332787 24602 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 8fcb2230014a476a8eb1eff1f6625fd2 to finish bootstrapping
I20260804 14:51:24.339588 24602 ts_itest-base.cc:233] Waiting for 1 tablets on tserver cdc5e32317de46d2ab6214138fd191e6 to finish bootstrapping
I20260804 14:51:24.371892 25800 raft_consensus.cc:1261] T 405aa67ad6d14cc5bacd40d24aa238d6 P 000ec79ca0274f6b83147bfc56d600ad [term 1 FOLLOWER]: Refusing update from remote peer 8fcb2230014a476a8eb1eff1f6625fd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260804 14:51:24.371973 26162 raft_consensus.cc:1261] T 405aa67ad6d14cc5bacd40d24aa238d6 P cdc5e32317de46d2ab6214138fd191e6 [term 1 FOLLOWER]: Refusing update from remote peer 8fcb2230014a476a8eb1eff1f6625fd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260804 14:51:24.372745 26271 consensus_queue.cc:1035] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdc5e32317de46d2ab6214138fd191e6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260804 14:51:24.372951 26266 consensus_queue.cc:1035] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "000ec79ca0274f6b83147bfc56d600ad" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260804 14:51:24.382858 26280 mvcc.cc:205] Tried to move back new op lower bound from 7314862425579110400 to 7314862425162260480. Current Snapshot: MvccSnapshot[applied={T|T < 7314862425579110400}]
I20260804 14:51:24.502614 25771 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260804 14:51:24.502647 26132 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260804 14:51:24.502647 25954 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
d2f5298df06d41c6b4954af3bc6f6984 | 127.24.6.190:39907 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.24.6.148:38921 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
000ec79ca0274f6b83147bfc56d600ad | 127.24.6.129:34809 | HEALTHY | <none> | 0 | 0
8fcb2230014a476a8eb1eff1f6625fd2 | 127.24.6.130:43911 | HEALTHY | <none> | 1 | 0
cdc5e32317de46d2ab6214138fd191e6 | 127.24.6.131:41355 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.24.6.129 | experimental | 127.24.6.129:34809
local_ip_for_outbound_sockets | 127.24.6.130 | experimental | 127.24.6.130:43911
local_ip_for_outbound_sockets | 127.24.6.131 | experimental | 127.24.6.131:41355
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.24.6.129:34809
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.24.6.130:43911
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.24.6.131:41355
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.24.6.148:38921 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260804 14:51:24.538375 24602 log_verifier.cc:126] Checking tablet 405aa67ad6d14cc5bacd40d24aa238d6
I20260804 14:51:24.572404 24602 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 405aa67ad6d14cc5bacd40d24aa238d6
I20260804 14:51:24.843240 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 25867
I20260804 14:51:24.862323 26162 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "405aa67ad6d14cc5bacd40d24aa238d6"
dest_uuid: "cdc5e32317de46d2ab6214138fd191e6"
from {username='slave'} at 127.0.0.1:33230
I20260804 14:51:24.862598 26162 raft_consensus.cc:482] T 405aa67ad6d14cc5bacd40d24aa238d6 P cdc5e32317de46d2ab6214138fd191e6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260804 14:51:24.862690 26162 raft_consensus.cc:3037] T 405aa67ad6d14cc5bacd40d24aa238d6 P cdc5e32317de46d2ab6214138fd191e6 [term 1 FOLLOWER]: Advancing to term 2
I20260804 14:51:24.863862 26162 raft_consensus.cc:504] T 405aa67ad6d14cc5bacd40d24aa238d6 P cdc5e32317de46d2ab6214138fd191e6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcb2230014a476a8eb1eff1f6625fd2" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43911 } } peers { permanent_uuid: "cdc5e32317de46d2ab6214138fd191e6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41355 } } peers { permanent_uuid: "000ec79ca0274f6b83147bfc56d600ad" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34809 } }
I20260804 14:51:24.865069 26162 leader_election.cc:288] T 405aa67ad6d14cc5bacd40d24aa238d6 P cdc5e32317de46d2ab6214138fd191e6 [CANDIDATE]: Term 2 election: Requested vote from peers 8fcb2230014a476a8eb1eff1f6625fd2 (127.24.6.130:43911), 000ec79ca0274f6b83147bfc56d600ad (127.24.6.129:34809)
W20260804 14:51:24.865850 26061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.6.130:43911: connect: Connection refused (error 111)
W20260804 14:51:24.867020 26061 leader_election.cc:334] T 405aa67ad6d14cc5bacd40d24aa238d6 P cdc5e32317de46d2ab6214138fd191e6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8fcb2230014a476a8eb1eff1f6625fd2 (127.24.6.130:43911): Network error: Client connection negotiation failed: client connection to 127.24.6.130:43911: connect: Connection refused (error 111)
I20260804 14:51:24.869843 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "405aa67ad6d14cc5bacd40d24aa238d6" candidate_uuid: "cdc5e32317de46d2ab6214138fd191e6" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "000ec79ca0274f6b83147bfc56d600ad"
I20260804 14:51:24.870002 25800 raft_consensus.cc:3037] T 405aa67ad6d14cc5bacd40d24aa238d6 P 000ec79ca0274f6b83147bfc56d600ad [term 1 FOLLOWER]: Advancing to term 2
I20260804 14:51:24.870980 25800 raft_consensus.cc:2454] T 405aa67ad6d14cc5bacd40d24aa238d6 P 000ec79ca0274f6b83147bfc56d600ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdc5e32317de46d2ab6214138fd191e6 in term 2.
I20260804 14:51:24.871412 26066 leader_election.cc:302] T 405aa67ad6d14cc5bacd40d24aa238d6 P cdc5e32317de46d2ab6214138fd191e6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 000ec79ca0274f6b83147bfc56d600ad, cdc5e32317de46d2ab6214138fd191e6; no voters: 8fcb2230014a476a8eb1eff1f6625fd2
I20260804 14:51:24.871742 26267 raft_consensus.cc:2781] T 405aa67ad6d14cc5bacd40d24aa238d6 P cdc5e32317de46d2ab6214138fd191e6 [term 2 FOLLOWER]: Leader election won for term 2
I20260804 14:51:24.871963 26267 raft_consensus.cc:686] T 405aa67ad6d14cc5bacd40d24aa238d6 P cdc5e32317de46d2ab6214138fd191e6 [term 2 LEADER]: Becoming Leader. State: Replica: cdc5e32317de46d2ab6214138fd191e6, State: Running, Role: LEADER
I20260804 14:51:24.872262 26267 consensus_queue.cc:237] T 405aa67ad6d14cc5bacd40d24aa238d6 P cdc5e32317de46d2ab6214138fd191e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcb2230014a476a8eb1eff1f6625fd2" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43911 } } peers { permanent_uuid: "cdc5e32317de46d2ab6214138fd191e6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41355 } } peers { permanent_uuid: "000ec79ca0274f6b83147bfc56d600ad" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34809 } }
W20260804 14:51:24.873214 26232 meta_cache.cc:302] tablet 405aa67ad6d14cc5bacd40d24aa238d6: replica 8fcb2230014a476a8eb1eff1f6625fd2 (127.24.6.130:43911) has failed: Network error: Client connection negotiation failed: client connection to 127.24.6.130:43911: connect: Connection refused (error 111)
I20260804 14:51:24.875183 25611 catalog_manager.cc:5495] T 405aa67ad6d14cc5bacd40d24aa238d6 P cdc5e32317de46d2ab6214138fd191e6 reported cstate change: term changed from 1 to 2, leader changed from 8fcb2230014a476a8eb1eff1f6625fd2 (127.24.6.130) to cdc5e32317de46d2ab6214138fd191e6 (127.24.6.131). New cstate: current_term: 2 leader_uuid: "cdc5e32317de46d2ab6214138fd191e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcb2230014a476a8eb1eff1f6625fd2" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdc5e32317de46d2ab6214138fd191e6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "000ec79ca0274f6b83147bfc56d600ad" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34809 } health_report { overall_health: UNKNOWN } } }
I20260804 14:51:24.886482 25800 raft_consensus.cc:1261] T 405aa67ad6d14cc5bacd40d24aa238d6 P 000ec79ca0274f6b83147bfc56d600ad [term 2 FOLLOWER]: Refusing update from remote peer cdc5e32317de46d2ab6214138fd191e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20260804 14:51:24.887028 26061 consensus_peers.cc:487] T 405aa67ad6d14cc5bacd40d24aa238d6 P cdc5e32317de46d2ab6214138fd191e6 -> Peer 8fcb2230014a476a8eb1eff1f6625fd2 (127.24.6.130:43911): Couldn't send request to peer 8fcb2230014a476a8eb1eff1f6625fd2. Status: Network error: Client connection negotiation failed: client connection to 127.24.6.130:43911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260804 14:51:24.887069 26267 consensus_queue.cc:1035] T 405aa67ad6d14cc5bacd40d24aa238d6 P cdc5e32317de46d2ab6214138fd191e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "000ec79ca0274f6b83147bfc56d600ad" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260804 14:51:24.932375 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.130:43911
--local_ip_for_outbound_sockets=127.24.6.130
--tserver_master_addrs=127.24.6.190:39907
--webserver_port=33335
--webserver_interface=127.24.6.130
--builtin_ntp_servers=127.24.6.148:38921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
W20260804 14:51:25.074124 26349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:51:25.074437 26349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:51:25.074476 26349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:51:25.078962 26349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:51:25.079074 26349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.130
I20260804 14:51:25.084108 26349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:38921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.130:43911
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.24.6.130
--webserver_port=33335
--tserver_master_addrs=127.24.6.190:39907
--never_fsync=true
--heap_profile_path=/tmp/kudu.26349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.130
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:51:25.085323 26349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:51:25.086791 26349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:51:25.097776 26360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:51:25.097884 26358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:51:25.097819 26355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:51:25.097970 26349 server_base.cc:1034] running on GCE node
I20260804 14:51:25.098716 26349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:51:25.099609 26349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:51:25.100834 26349 hybrid_clock.cc:648] HybridClock initialized: now 1785855085100822 us; error 44 us; skew 500 ppm
I20260804 14:51:25.102902 26349 webserver.cc:460] Webserver started at http://127.24.6.130:33335/ using document root <none> and password file <none>
I20260804 14:51:25.103617 26349 fs_manager.cc:359] Metadata directory not provided
I20260804 14:51:25.103744 26349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:51:25.108120 26349 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260804 14:51:25.110972 26367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:51:25.112407 26349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260804 14:51:25.112684 26349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "8fcb2230014a476a8eb1eff1f6625fd2"
format_stamp: "Formatted at 2026-08-04 14:51:23 on dist-test-slave-92m1"
I20260804 14:51:25.113286 26349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:51:25.139954 26349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:51:25.140764 26349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:51:25.140985 26349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:51:25.141726 26349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:51:25.143085 26374 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260804 14:51:25.147392 26349 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260804 14:51:25.147506 26349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260804 14:51:25.147603 26349 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260804 14:51:25.149299 26349 ts_tablet_manager.cc:610] Registered 1 tablets
I20260804 14:51:25.149377 26349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260804 14:51:25.149665 26374 tablet_bootstrap.cc:492] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2: Bootstrap starting.
I20260804 14:51:25.162323 26349 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.130:43911
I20260804 14:51:25.162400 26513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.130:43911 every 8 connection(s)
I20260804 14:51:25.163524 26349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260804 14:51:25.165741 26374 log.cc:824] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2: Log is configured to *not* fsync() on all Append() calls
I20260804 14:51:25.172122 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 26349
I20260804 14:51:25.175925 26514 heartbeater.cc:347] Connected to a master server at 127.24.6.190:39907
I20260804 14:51:25.176182 26514 heartbeater.cc:464] Registering TS with master...
I20260804 14:51:25.176793 26514 heartbeater.cc:511] Master 127.24.6.190:39907 requested a full tablet report, sending...
I20260804 14:51:25.178987 25611 ts_manager.cc:194] Re-registered known tserver with Master: 8fcb2230014a476a8eb1eff1f6625fd2 (127.24.6.130:43911)
I20260804 14:51:25.180081 25611 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.130:45525
I20260804 14:51:25.189811 26374 tablet_bootstrap.cc:492] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=500 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260804 14:51:25.190296 26374 tablet_bootstrap.cc:492] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2: Bootstrap complete.
I20260804 14:51:25.191568 26374 ts_tablet_manager.cc:1397] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2: Time spent bootstrapping tablet: real 0.042s user 0.036s sys 0.004s
I20260804 14:51:25.193992 26374 raft_consensus.cc:348] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcb2230014a476a8eb1eff1f6625fd2" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43911 } } peers { permanent_uuid: "cdc5e32317de46d2ab6214138fd191e6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41355 } } peers { permanent_uuid: "000ec79ca0274f6b83147bfc56d600ad" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34809 } }
I20260804 14:51:25.194413 26374 raft_consensus.cc:729] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fcb2230014a476a8eb1eff1f6625fd2, State: Initialized, Role: FOLLOWER
I20260804 14:51:25.194902 26374 consensus_queue.cc:260] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcb2230014a476a8eb1eff1f6625fd2" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43911 } } peers { permanent_uuid: "cdc5e32317de46d2ab6214138fd191e6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41355 } } peers { permanent_uuid: "000ec79ca0274f6b83147bfc56d600ad" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34809 } }
I20260804 14:51:25.195693 26514 heartbeater.cc:503] Master 127.24.6.190:39907 was elected leader, sending a full tablet report...
I20260804 14:51:25.196034 26374 ts_tablet_manager.cc:1428] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260804 14:51:25.229329 25771 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260804 14:51:25.230866 26132 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260804 14:51:25.233603 26424 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
d2f5298df06d41c6b4954af3bc6f6984 | 127.24.6.190:39907 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.24.6.148:38921 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
000ec79ca0274f6b83147bfc56d600ad | 127.24.6.129:34809 | HEALTHY | <none> | 0 | 0
8fcb2230014a476a8eb1eff1f6625fd2 | 127.24.6.130:43911 | HEALTHY | <none> | 0 | 0
cdc5e32317de46d2ab6214138fd191e6 | 127.24.6.131:41355 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.24.6.129 | experimental | 127.24.6.129:34809
local_ip_for_outbound_sockets | 127.24.6.130 | experimental | 127.24.6.130:43911
local_ip_for_outbound_sockets | 127.24.6.131 | experimental | 127.24.6.131:41355
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.24.6.129:34809
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.24.6.130:43911
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.24.6.131:41355
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.24.6.148:38921 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 405aa67ad6d14cc5bacd40d24aa238d6 of table 'TestTable' is conflicted: 1 replicas' active configs disagree with the leader master's
8fcb2230014a476a8eb1eff1f6625fd2 (127.24.6.130:43911): RUNNING
cdc5e32317de46d2ab6214138fd191e6 (127.24.6.131:41355): RUNNING [LEADER]
000ec79ca0274f6b83147bfc56d600ad (127.24.6.129:34809): RUNNING
All reported replicas are:
A = 8fcb2230014a476a8eb1eff1f6625fd2
B = cdc5e32317de46d2ab6214138fd191e6
C = 000ec79ca0274f6b83147bfc56d600ad
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B C | 1 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | A B* C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+--------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260804 14:51:25.256206 24602 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260804 14:51:25.450778 26449 raft_consensus.cc:3037] T 405aa67ad6d14cc5bacd40d24aa238d6 P 8fcb2230014a476a8eb1eff1f6625fd2 [term 1 FOLLOWER]: Advancing to term 2
I20260804 14:51:25.464638 25771 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260804 14:51:25.465358 26424 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260804 14:51:25.465739 26132 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260804 14:51:25.469929 26588 mvcc.cc:205] Tried to move back new op lower bound from 7314862427863347200 to 7314862427640139776. Current Snapshot: MvccSnapshot[applied={T|T < 7314862427689070592}]
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
d2f5298df06d41c6b4954af3bc6f6984 | 127.24.6.190:39907 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.24.6.148:38921 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
000ec79ca0274f6b83147bfc56d600ad | 127.24.6.129:34809 | HEALTHY | <none> | 0 | 0
8fcb2230014a476a8eb1eff1f6625fd2 | 127.24.6.130:43911 | HEALTHY | <none> | 0 | 0
cdc5e32317de46d2ab6214138fd191e6 | 127.24.6.131:41355 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.24.6.129 | experimental | 127.24.6.129:34809
local_ip_for_outbound_sockets | 127.24.6.130 | experimental | 127.24.6.130:43911
local_ip_for_outbound_sockets | 127.24.6.131 | experimental | 127.24.6.131:41355
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.24.6.129:34809
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.24.6.130:43911
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.24.6.131:41355
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.24.6.148:38921 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260804 14:51:25.489889 24602 log_verifier.cc:126] Checking tablet 405aa67ad6d14cc5bacd40d24aa238d6
I20260804 14:51:25.513814 24602 log_verifier.cc:177] Verified matching terms for 12 ops in tablet 405aa67ad6d14cc5bacd40d24aa238d6
I20260804 14:51:25.531888 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 25695
I20260804 14:51:25.545389 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 26349
I20260804 14:51:25.562350 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 26041
I20260804 14:51:25.576862 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 25562
2026-08-04T14:51:25Z chronyd exiting
[ OK ] RaftConsensusElectionITest.RunLeaderElection (2683 ms)
[ RUN ] RaftConsensusElectionITest.ChurnyElections
I20260804 14:51:25.608888 24602 test_util.cc:268] Using random seed: -906002510
I20260804 14:51:25.609807 24602 ts_itest-base.cc:112] Starting cluster with:
I20260804 14:51:25.609853 24602 ts_itest-base.cc:113] --------------
I20260804 14:51:25.609866 24602 ts_itest-base.cc:114] 3 tablet servers
I20260804 14:51:25.609879 24602 ts_itest-base.cc:115] 3 replicas per TS
I20260804 14:51:25.609891 24602 ts_itest-base.cc:116] --------------
2026-08-04T14:51:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-04T14:51:25Z Disabled control of system clock
I20260804 14:51:25.622515 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.6.190:43405
--webserver_interface=127.24.6.190
--webserver_port=0
--builtin_ntp_servers=127.24.6.148:38557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.6.190:43405 with env {}
W20260804 14:51:25.745538 26612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:51:25.745908 26612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:51:25.746001 26612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:51:25.750373 26612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260804 14:51:25.750483 26612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:51:25.750517 26612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260804 14:51:25.750558 26612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260804 14:51:25.755786 26612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:38557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.6.190:43405
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.6.190:43405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.24.6.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.26612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:51:25.757118 26612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:51:25.758657 26612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:51:25.768260 26618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:51:25.768361 26612 server_base.cc:1034] running on GCE node
W20260804 14:51:25.768213 26619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:51:25.768235 26622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:51:25.769268 26612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:51:25.770545 26612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:51:25.771773 26612 hybrid_clock.cc:648] HybridClock initialized: now 1785855085771754 us; error 38 us; skew 500 ppm
I20260804 14:51:25.774188 26612 webserver.cc:460] Webserver started at http://127.24.6.190:38111/ using document root <none> and password file <none>
I20260804 14:51:25.774869 26612 fs_manager.cc:359] Metadata directory not provided
I20260804 14:51:25.774976 26612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:51:25.775295 26612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:51:25.777349 26612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "862a2eb595284226b154fa6290ec0ce2"
format_stamp: "Formatted at 2026-08-04 14:51:25 on dist-test-slave-92m1"
I20260804 14:51:25.777957 26612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "862a2eb595284226b154fa6290ec0ce2"
format_stamp: "Formatted at 2026-08-04 14:51:25 on dist-test-slave-92m1"
I20260804 14:51:25.782150 26612 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260804 14:51:25.785071 26629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:51:25.786453 26612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260804 14:51:25.786640 26612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
uuid: "862a2eb595284226b154fa6290ec0ce2"
format_stamp: "Formatted at 2026-08-04 14:51:25 on dist-test-slave-92m1"
I20260804 14:51:25.786777 26612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:51:25.825645 26612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:51:25.826510 26612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:51:25.826749 26612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:51:25.834669 26612 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.190:43405
I20260804 14:51:25.834676 26701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.190:43405 every 8 connection(s)
I20260804 14:51:25.835799 26612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260804 14:51:25.839919 26702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:51:25.840791 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 26612
I20260804 14:51:25.840955 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260804 14:51:25.846017 26702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 862a2eb595284226b154fa6290ec0ce2: Bootstrap starting.
I20260804 14:51:25.848624 26702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 862a2eb595284226b154fa6290ec0ce2: Neither blocks nor log segments found. Creating new log.
I20260804 14:51:25.849633 26702 log.cc:824] T 00000000000000000000000000000000 P 862a2eb595284226b154fa6290ec0ce2: Log is configured to *not* fsync() on all Append() calls
I20260804 14:51:25.852175 26702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 862a2eb595284226b154fa6290ec0ce2: No bootstrap required, opened a new log
I20260804 14:51:25.856091 26702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 862a2eb595284226b154fa6290ec0ce2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862a2eb595284226b154fa6290ec0ce2" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 43405 } }
I20260804 14:51:25.856393 26702 raft_consensus.cc:374] T 00000000000000000000000000000000 P 862a2eb595284226b154fa6290ec0ce2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:51:25.856482 26702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 862a2eb595284226b154fa6290ec0ce2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862a2eb595284226b154fa6290ec0ce2, State: Initialized, Role: FOLLOWER
I20260804 14:51:25.856949 26702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 862a2eb595284226b154fa6290ec0ce2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862a2eb595284226b154fa6290ec0ce2" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 43405 } }
I20260804 14:51:25.857115 26702 raft_consensus.cc:388] T 00000000000000000000000000000000 P 862a2eb595284226b154fa6290ec0ce2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260804 14:51:25.857208 26702 raft_consensus.cc:482] T 00000000000000000000000000000000 P 862a2eb595284226b154fa6290ec0ce2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260804 14:51:25.857332 26702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 862a2eb595284226b154fa6290ec0ce2 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:51:25.858502 26702 raft_consensus.cc:504] T 00000000000000000000000000000000 P 862a2eb595284226b154fa6290ec0ce2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862a2eb595284226b154fa6290ec0ce2" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 43405 } }
I20260804 14:51:25.858872 26702 leader_election.cc:302] T 00000000000000000000000000000000 P 862a2eb595284226b154fa6290ec0ce2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 862a2eb595284226b154fa6290ec0ce2; no voters:
I20260804 14:51:25.859261 26702 leader_election.cc:288] T 00000000000000000000000000000000 P 862a2eb595284226b154fa6290ec0ce2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260804 14:51:25.859441 26707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 862a2eb595284226b154fa6290ec0ce2 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:51:25.859707 26707 raft_consensus.cc:686] T 00000000000000000000000000000000 P 862a2eb595284226b154fa6290ec0ce2 [term 1 LEADER]: Becoming Leader. State: Replica: 862a2eb595284226b154fa6290ec0ce2, State: Running, Role: LEADER
I20260804 14:51:25.860082 26707 consensus_queue.cc:237] T 00000000000000000000000000000000 P 862a2eb595284226b154fa6290ec0ce2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862a2eb595284226b154fa6290ec0ce2" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 43405 } }
I20260804 14:51:25.860704 26702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 862a2eb595284226b154fa6290ec0ce2 [sys.catalog]: configured and running, proceeding with master startup.
I20260804 14:51:25.862052 26708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 862a2eb595284226b154fa6290ec0ce2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "862a2eb595284226b154fa6290ec0ce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862a2eb595284226b154fa6290ec0ce2" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 43405 } } }
I20260804 14:51:25.862104 26709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 862a2eb595284226b154fa6290ec0ce2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 862a2eb595284226b154fa6290ec0ce2. Latest consensus state: current_term: 1 leader_uuid: "862a2eb595284226b154fa6290ec0ce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862a2eb595284226b154fa6290ec0ce2" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 43405 } } }
I20260804 14:51:25.862193 26708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 862a2eb595284226b154fa6290ec0ce2 [sys.catalog]: This master's current role is: LEADER
I20260804 14:51:25.862210 26709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 862a2eb595284226b154fa6290ec0ce2 [sys.catalog]: This master's current role is: LEADER
I20260804 14:51:25.862560 26721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260804 14:51:25.866217 26721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260804 14:51:25.873327 26721 catalog_manager.cc:1347] Generated new cluster ID: c593f41132ff4a648cc06fa1f18eeeb3
I20260804 14:51:25.873464 26721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260804 14:51:25.895592 26721 catalog_manager.cc:1370] Generated new certificate authority record
I20260804 14:51:25.897199 26721 catalog_manager.cc:1504] Loading token signing keys...
I20260804 14:51:25.902163 26721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 862a2eb595284226b154fa6290ec0ce2: Generated new TSK 0
I20260804 14:51:25.902916 26721 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260804 14:51:25.907823 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.129:0
--local_ip_for_outbound_sockets=127.24.6.129
--webserver_interface=127.24.6.129
--webserver_port=0
--tserver_master_addrs=127.24.6.190:43405
--builtin_ntp_servers=127.24.6.148:38557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
W20260804 14:51:26.059533 26736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:51:26.059885 26736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:51:26.059968 26736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:51:26.064518 26736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:51:26.064648 26736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.129
I20260804 14:51:26.069654 26736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:38557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.24.6.129
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.24.6.190:43405
--never_fsync=true
--heap_profile_path=/tmp/kudu.26736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.129
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:51:26.070886 26736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:51:26.072484 26736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:51:26.082988 26746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:51:26.083148 26736 server_base.cc:1034] running on GCE node
W20260804 14:51:26.083024 26748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:51:26.083376 26743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:51:26.083848 26736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:51:26.084621 26736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:51:26.085851 26736 hybrid_clock.cc:648] HybridClock initialized: now 1785855086085809 us; error 49 us; skew 500 ppm
I20260804 14:51:26.088189 26736 webserver.cc:460] Webserver started at http://127.24.6.129:38979/ using document root <none> and password file <none>
I20260804 14:51:26.088891 26736 fs_manager.cc:359] Metadata directory not provided
I20260804 14:51:26.089004 26736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:51:26.089277 26736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:51:26.091794 26736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "a2bfdd3e90e148468b3755c42cad8856"
format_stamp: "Formatted at 2026-08-04 14:51:26 on dist-test-slave-92m1"
I20260804 14:51:26.092519 26736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "a2bfdd3e90e148468b3755c42cad8856"
format_stamp: "Formatted at 2026-08-04 14:51:26 on dist-test-slave-92m1"
I20260804 14:51:26.097152 26736 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260804 14:51:26.100159 26754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:51:26.101577 26736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260804 14:51:26.101738 26736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "a2bfdd3e90e148468b3755c42cad8856"
format_stamp: "Formatted at 2026-08-04 14:51:26 on dist-test-slave-92m1"
I20260804 14:51:26.101874 26736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:51:26.124655 26736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:51:26.125550 26736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:51:26.125795 26736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:51:26.126515 26736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:51:26.127707 26736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:51:26.127787 26736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:51:26.127872 26736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:51:26.127931 26736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:51:26.138033 26736 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.129:42087
I20260804 14:51:26.138082 26899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.129:42087 every 8 connection(s)
I20260804 14:51:26.139137 26736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260804 14:51:26.147923 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 26736
I20260804 14:51:26.148094 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260804 14:51:26.150831 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.130:0
--local_ip_for_outbound_sockets=127.24.6.130
--webserver_interface=127.24.6.130
--webserver_port=0
--tserver_master_addrs=127.24.6.190:43405
--builtin_ntp_servers=127.24.6.148:38557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
I20260804 14:51:26.151046 26903 heartbeater.cc:347] Connected to a master server at 127.24.6.190:43405
I20260804 14:51:26.151383 26903 heartbeater.cc:464] Registering TS with master...
I20260804 14:51:26.152125 26903 heartbeater.cc:511] Master 127.24.6.190:43405 requested a full tablet report, sending...
I20260804 14:51:26.154270 26657 ts_manager.cc:194] Registered new tserver with Master: a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087)
I20260804 14:51:26.155682 26657 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.129:50431
W20260804 14:51:26.273136 26906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:51:26.273497 26906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:51:26.273569 26906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:51:26.277961 26906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:51:26.278174 26906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.130
I20260804 14:51:26.283443 26906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:38557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.24.6.130
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.24.6.190:43405
--never_fsync=true
--heap_profile_path=/tmp/kudu.26906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.130
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:51:26.284971 26906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:51:26.286607 26906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260804 14:51:26.297290 26906 server_base.cc:1034] running on GCE node
W20260804 14:51:26.297212 26912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:51:26.297235 26916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:51:26.297237 26914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:51:26.297947 26906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:51:26.298656 26906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:51:26.299849 26906 hybrid_clock.cc:648] HybridClock initialized: now 1785855086299811 us; error 49 us; skew 500 ppm
I20260804 14:51:26.301834 26906 webserver.cc:460] Webserver started at http://127.24.6.130:40635/ using document root <none> and password file <none>
I20260804 14:51:26.302474 26906 fs_manager.cc:359] Metadata directory not provided
I20260804 14:51:26.302572 26906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:51:26.302822 26906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:51:26.305042 26906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "9105c783d9e14fa3af410b705c5daebd"
format_stamp: "Formatted at 2026-08-04 14:51:26 on dist-test-slave-92m1"
I20260804 14:51:26.305685 26906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "9105c783d9e14fa3af410b705c5daebd"
format_stamp: "Formatted at 2026-08-04 14:51:26 on dist-test-slave-92m1"
I20260804 14:51:26.310114 26906 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260804 14:51:26.313334 26926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:51:26.314795 26906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260804 14:51:26.314949 26906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "9105c783d9e14fa3af410b705c5daebd"
format_stamp: "Formatted at 2026-08-04 14:51:26 on dist-test-slave-92m1"
I20260804 14:51:26.315085 26906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:51:26.341754 26906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:51:26.342607 26906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:51:26.342844 26906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:51:26.343564 26906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:51:26.344666 26906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:51:26.344743 26906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:51:26.344821 26906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:51:26.344866 26906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:51:26.355134 26906 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.130:37489
I20260804 14:51:26.355154 27078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.130:37489 every 8 connection(s)
I20260804 14:51:26.356240 26906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260804 14:51:26.361083 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 26906
I20260804 14:51:26.361243 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260804 14:51:26.363986 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.131:0
--local_ip_for_outbound_sockets=127.24.6.131
--webserver_interface=127.24.6.131
--webserver_port=0
--tserver_master_addrs=127.24.6.190:43405
--builtin_ntp_servers=127.24.6.148:38557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
I20260804 14:51:26.369005 27079 heartbeater.cc:347] Connected to a master server at 127.24.6.190:43405
I20260804 14:51:26.369310 27079 heartbeater.cc:464] Registering TS with master...
I20260804 14:51:26.369920 27079 heartbeater.cc:511] Master 127.24.6.190:43405 requested a full tablet report, sending...
I20260804 14:51:26.371294 26657 ts_manager.cc:194] Registered new tserver with Master: 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:26.371934 26657 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.130:54389
W20260804 14:51:26.494148 27083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:51:26.494536 27083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:51:26.494621 27083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:51:26.500821 27083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:51:26.501073 27083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.131
I20260804 14:51:26.509101 27083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:38557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.24.6.131
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.24.6.190:43405
--never_fsync=true
--heap_profile_path=/tmp/kudu.27083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.131
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:51:26.510690 27083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:51:26.512826 27083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:51:26.527850 27089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:51:26.527813 27091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:51:26.528244 27083 server_base.cc:1034] running on GCE node
W20260804 14:51:26.528460 27097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:51:26.529035 27083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:51:26.529801 27083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:51:26.531028 27083 hybrid_clock.cc:648] HybridClock initialized: now 1785855086530975 us; error 59 us; skew 500 ppm
I20260804 14:51:26.533540 27083 webserver.cc:460] Webserver started at http://127.24.6.131:42169/ using document root <none> and password file <none>
I20260804 14:51:26.534227 27083 fs_manager.cc:359] Metadata directory not provided
I20260804 14:51:26.534339 27083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:51:26.534601 27083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:51:26.536922 27083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "59538ca2ea484ee197ada4dbcbeac3bf"
format_stamp: "Formatted at 2026-08-04 14:51:26 on dist-test-slave-92m1"
I20260804 14:51:26.537588 27083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "59538ca2ea484ee197ada4dbcbeac3bf"
format_stamp: "Formatted at 2026-08-04 14:51:26 on dist-test-slave-92m1"
I20260804 14:51:26.542956 27083 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260804 14:51:26.545874 27105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:51:26.547308 27083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260804 14:51:26.547456 27083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "59538ca2ea484ee197ada4dbcbeac3bf"
format_stamp: "Formatted at 2026-08-04 14:51:26 on dist-test-slave-92m1"
I20260804 14:51:26.547592 27083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:51:26.560731 27083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:51:26.561635 27083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:51:26.561843 27083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:51:26.562536 27083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:51:26.563678 27083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:51:26.563763 27083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:51:26.563838 27083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:51:26.563895 27083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:51:26.573526 27083 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.131:42021
I20260804 14:51:26.573555 27262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.131:42021 every 8 connection(s)
I20260804 14:51:26.574558 27083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260804 14:51:26.583675 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 27083
I20260804 14:51:26.583827 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260804 14:51:26.586493 27263 heartbeater.cc:347] Connected to a master server at 127.24.6.190:43405
I20260804 14:51:26.586810 27263 heartbeater.cc:464] Registering TS with master...
I20260804 14:51:26.587473 27263 heartbeater.cc:511] Master 127.24.6.190:43405 requested a full tablet report, sending...
I20260804 14:51:26.588587 26655 ts_manager.cc:194] Registered new tserver with Master: 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:26.589231 26655 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.131:43075
I20260804 14:51:26.599031 24602 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260804 14:51:26.600930 24602 test_util.cc:268] Using random seed: -905010466
I20260804 14:51:26.614507 26655 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50476:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260804 14:51:26.615869 26655 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260804 14:51:26.635483 26819 tablet_service.cc:1467] Processing CreateTablet for tablet df07e91451bc49ba978359f97a315ebe (DEFAULT_TABLE table=test-workload [id=c1e5b5fefcdb4c048d0b1a9f52078c02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:51:26.635588 26994 tablet_service.cc:1467] Processing CreateTablet for tablet df07e91451bc49ba978359f97a315ebe (DEFAULT_TABLE table=test-workload [id=c1e5b5fefcdb4c048d0b1a9f52078c02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:51:26.635577 27174 tablet_service.cc:1467] Processing CreateTablet for tablet df07e91451bc49ba978359f97a315ebe (DEFAULT_TABLE table=test-workload [id=c1e5b5fefcdb4c048d0b1a9f52078c02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:51:26.636631 26819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df07e91451bc49ba978359f97a315ebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:51:26.636657 27174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df07e91451bc49ba978359f97a315ebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:51:26.636657 26994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df07e91451bc49ba978359f97a315ebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:51:26.644023 27293 tablet_bootstrap.cc:492] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf: Bootstrap starting.
I20260804 14:51:26.644384 27292 tablet_bootstrap.cc:492] T df07e91451bc49ba978359f97a315ebe P 9105c783d9e14fa3af410b705c5daebd: Bootstrap starting.
I20260804 14:51:26.646445 27292 tablet_bootstrap.cc:654] T df07e91451bc49ba978359f97a315ebe P 9105c783d9e14fa3af410b705c5daebd: Neither blocks nor log segments found. Creating new log.
I20260804 14:51:26.646445 27293 tablet_bootstrap.cc:654] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf: Neither blocks nor log segments found. Creating new log.
I20260804 14:51:26.647058 27291 tablet_bootstrap.cc:492] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856: Bootstrap starting.
I20260804 14:51:26.647390 27293 log.cc:824] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf: Log is configured to *not* fsync() on all Append() calls
I20260804 14:51:26.647385 27292 log.cc:824] T df07e91451bc49ba978359f97a315ebe P 9105c783d9e14fa3af410b705c5daebd: Log is configured to *not* fsync() on all Append() calls
I20260804 14:51:26.649605 27291 tablet_bootstrap.cc:654] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856: Neither blocks nor log segments found. Creating new log.
I20260804 14:51:26.649739 27293 tablet_bootstrap.cc:492] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf: No bootstrap required, opened a new log
I20260804 14:51:26.649957 27293 ts_tablet_manager.cc:1397] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260804 14:51:26.650502 27291 log.cc:824] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856: Log is configured to *not* fsync() on all Append() calls
I20260804 14:51:26.650545 27292 tablet_bootstrap.cc:492] T df07e91451bc49ba978359f97a315ebe P 9105c783d9e14fa3af410b705c5daebd: No bootstrap required, opened a new log
I20260804 14:51:26.650766 27292 ts_tablet_manager.cc:1397] T df07e91451bc49ba978359f97a315ebe P 9105c783d9e14fa3af410b705c5daebd: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260804 14:51:26.653065 27291 tablet_bootstrap.cc:492] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856: No bootstrap required, opened a new log
I20260804 14:51:26.653276 27291 ts_tablet_manager.cc:1397] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260804 14:51:26.654798 27292 raft_consensus.cc:348] T df07e91451bc49ba978359f97a315ebe P 9105c783d9e14fa3af410b705c5daebd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:26.654729 27293 raft_consensus.cc:348] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:26.655053 27292 raft_consensus.cc:374] T df07e91451bc49ba978359f97a315ebe P 9105c783d9e14fa3af410b705c5daebd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:51:26.655053 27293 raft_consensus.cc:374] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:51:26.655117 27292 raft_consensus.cc:729] T df07e91451bc49ba978359f97a315ebe P 9105c783d9e14fa3af410b705c5daebd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9105c783d9e14fa3af410b705c5daebd, State: Initialized, Role: FOLLOWER
I20260804 14:51:26.655117 27293 raft_consensus.cc:729] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59538ca2ea484ee197ada4dbcbeac3bf, State: Initialized, Role: FOLLOWER
I20260804 14:51:26.655663 27293 consensus_queue.cc:260] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:26.655632 27292 consensus_queue.cc:260] T df07e91451bc49ba978359f97a315ebe P 9105c783d9e14fa3af410b705c5daebd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:26.656870 27293 ts_tablet_manager.cc:1428] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260804 14:51:26.657263 27263 heartbeater.cc:503] Master 127.24.6.190:43405 was elected leader, sending a full tablet report...
I20260804 14:51:26.657465 27292 ts_tablet_manager.cc:1428] T df07e91451bc49ba978359f97a315ebe P 9105c783d9e14fa3af410b705c5daebd: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260804 14:51:26.657842 27079 heartbeater.cc:503] Master 127.24.6.190:43405 was elected leader, sending a full tablet report...
I20260804 14:51:26.657951 27306 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 9105c783d9e14fa3af410b705c5daebd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260804 14:51:26.657868 27291 raft_consensus.cc:348] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:26.658119 27291 raft_consensus.cc:374] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:51:26.658090 27306 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 9105c783d9e14fa3af410b705c5daebd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:26.658182 27291 raft_consensus.cc:729] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2bfdd3e90e148468b3755c42cad8856, State: Initialized, Role: FOLLOWER
I20260804 14:51:26.658670 27291 consensus_queue.cc:260] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:26.659391 27306 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 9105c783d9e14fa3af410b705c5daebd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:26.659775 27307 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260804 14:51:26.659976 27307 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:26.661239 27307 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:26.663725 27291 ts_tablet_manager.cc:1428] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260804 14:51:26.664206 27309 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260804 14:51:26.664266 26903 heartbeater.cc:503] Master 127.24.6.190:43405 was elected leader, sending a full tablet report...
I20260804 14:51:26.664368 27309 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
W20260804 14:51:26.665602 26929 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.24.6.129:42087 timed out after 0.006s (ON_OUTBOUND_QUEUE)
W20260804 14:51:26.668977 27108 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.24.6.129:42087 for RPC RequestConsensusVote timed out after 0.007s (ON_OUTBOUND_QUEUE)
I20260804 14:51:26.669049 27309 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:26.670365 26845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "9105c783d9e14fa3af410b705c5daebd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:26.670637 26845 raft_consensus.cc:2454] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9105c783d9e14fa3af410b705c5daebd in term 0.
I20260804 14:51:26.671119 26929 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 9105c783d9e14fa3af410b705c5daebd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856; no voters:
I20260804 14:51:26.671506 27305 raft_consensus.cc:2781] T df07e91451bc49ba978359f97a315ebe P 9105c783d9e14fa3af410b705c5daebd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260804 14:51:26.671598 27305 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 9105c783d9e14fa3af410b705c5daebd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260804 14:51:26.671641 27305 raft_consensus.cc:3037] T df07e91451bc49ba978359f97a315ebe P 9105c783d9e14fa3af410b705c5daebd [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:51:26.672788 27199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "9105c783d9e14fa3af410b705c5daebd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:26.673055 27199 raft_consensus.cc:2454] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9105c783d9e14fa3af410b705c5daebd in term 0.
I20260804 14:51:26.673100 27305 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 9105c783d9e14fa3af410b705c5daebd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:26.673524 27305 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 9105c783d9e14fa3af410b705c5daebd [CANDIDATE]: Term 1 election: Requested vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:26.673987 26845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "9105c783d9e14fa3af410b705c5daebd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856"
I20260804 14:51:26.674125 27197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:26.674158 26845 raft_consensus.cc:3037] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:51:26.674270 27197 raft_consensus.cc:2454] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2bfdd3e90e148468b3755c42cad8856 in term 0.
I20260804 14:51:26.674578 27199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "9105c783d9e14fa3af410b705c5daebd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf"
I20260804 14:51:26.674652 26755 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf, a2bfdd3e90e148468b3755c42cad8856; no voters:
I20260804 14:51:26.674695 27199 raft_consensus.cc:3037] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:51:26.675490 26845 raft_consensus.cc:2454] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9105c783d9e14fa3af410b705c5daebd in term 1.
I20260804 14:51:26.675709 27309 raft_consensus.cc:2741] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260804 14:51:26.675908 26929 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 9105c783d9e14fa3af410b705c5daebd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856; no voters:
I20260804 14:51:26.676044 27199 raft_consensus.cc:2454] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9105c783d9e14fa3af410b705c5daebd in term 1.
I20260804 14:51:26.676072 27305 raft_consensus.cc:2781] T df07e91451bc49ba978359f97a315ebe P 9105c783d9e14fa3af410b705c5daebd [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:51:26.676187 27305 raft_consensus.cc:686] T df07e91451bc49ba978359f97a315ebe P 9105c783d9e14fa3af410b705c5daebd [term 1 LEADER]: Becoming Leader. State: Replica: 9105c783d9e14fa3af410b705c5daebd, State: Running, Role: LEADER
I20260804 14:51:26.676586 27305 consensus_queue.cc:237] T df07e91451bc49ba978359f97a315ebe P 9105c783d9e14fa3af410b705c5daebd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
W20260804 14:51:26.677268 26757 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.24.6.130:37489 for RPC RequestConsensusVote timed out after 0.007s (ON_OUTBOUND_QUEUE)
I20260804 14:51:26.677292 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:26.677588 26845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:26.677764 26845 raft_consensus.cc:2379] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59538ca2ea484ee197ada4dbcbeac3bf in current term 1: Already voted for candidate 9105c783d9e14fa3af410b705c5daebd in this term.
I20260804 14:51:26.678208 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:26.679045 27302 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260804 14:51:26.680897 26657 catalog_manager.cc:5495] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9105c783d9e14fa3af410b705c5daebd (127.24.6.130). New cstate: current_term: 1 leader_uuid: "9105c783d9e14fa3af410b705c5daebd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } } }
I20260804 14:51:26.681514 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
W20260804 14:51:26.707261 27328 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260804 14:51:26.707350 27328 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20260804 14:51:26.789816 27199 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260804 14:51:26.789870 26845 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260804 14:51:26.789909 27199 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20260804 14:51:26.789934 26845 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260804 14:51:26.817561 27309 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:26.817728 27309 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:26.818661 27309 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:26.819105 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:26.819763 27197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:26.820370 26755 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:26.820624 27309 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:27.095584 27302 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:27.095745 27302 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:27.096639 27302 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:27.097091 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:27.097110 26845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:27.097748 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:27.097996 27302 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:27.144727 27309 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:27.144857 27309 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:27.145236 27309 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:27.145548 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:27.145567 27199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:27.145946 26755 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:27.146127 27309 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:27.154778 27309 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:27.154904 27309 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:27.155464 27309 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:27.155840 27197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:27.155879 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:27.156394 26757 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:27.156595 27309 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:27.320977 27302 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:27.321177 27302 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:27.321825 27302 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:27.322299 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:27.322293 26844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:27.322849 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:27.323117 27302 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:27.360432 27302 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:27.360598 27302 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:27.361076 27302 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:27.361904 26844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:27.362057 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:27.362596 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:27.362880 27302 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:27.548655 27302 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:27.548810 27302 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:27.549260 27302 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:27.550534 26844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:27.550532 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:27.551031 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:27.551244 27302 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:28.122189 27309 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:28.122352 27309 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:28.122889 27309 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:28.123310 27197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:28.123333 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:28.124297 26755 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:28.124509 27309 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:28.347270 27309 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:28.347504 27309 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:28.348047 27309 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:28.348483 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:28.349033 27199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:28.349186 27357 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50938 (ReqId={client: 2f11fb73142a45a5ab6d1317d6b28d6e, seq_no=2, attempt_no=0}) took 1526 ms. Trace:
I20260804 14:51:28.349862 26755 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:28.350116 27309 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:28.349272 27357 rpcz_store.cc:276] 0804 14:51:26.822962 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:26.823031 (+ 69us) service_pool.cc:227] Handling call
0804 14:51:28.349170 (+1526139us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:27.296631 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:27.296743 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:27.296748 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:51:27.296750 (+ 2us) tablet.cc:660] Decoding operations
0804 14:51:27.296860 (+ 110us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:27.296872 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:27.296875 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:27.296934 (+ 59us) tablet.cc:699] Row locks acquired
0804 14:51:27.296937 (+ 3us) write_op.cc:260] PREPARE: finished
0804 14:51:27.296985 (+ 48us) write_op.cc:270] Start()
0804 14:51:28.194144 (+897159us) write_op.cc:276] Timestamp: P: 1785855087296919 usec, L: 0
0804 14:51:28.194148 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:51:28.194478 (+ 330us) log.cc:842] Serialized 354 byte log entry
0804 14:51:28.348379 (+153901us) op_driver.cc:444] REPLICATION: finished
0804 14:51:28.348694 (+ 315us) write_op.cc:301] APPLY: starting
0804 14:51:28.348714 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0804 14:51:28.348751 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0804 14:51:28.348768 (+ 17us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:51:28.348857 (+ 89us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:51:28.348895 (+ 38us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:51:28.348900 (+ 5us) write_op.cc:312] APPLY: finished
0804 14:51:28.348974 (+ 74us) log.cc:842] Serialized 22 byte log entry
0804 14:51:28.349069 (+ 95us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:28.349085 (+ 16us) write_op.cc:454] Released schema lock
0804 14:51:28.349091 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":262,"num_ops":1,"prepare.queue_time_us":473142,"prepare.run_cpu_time_us":866,"prepare.run_wall_time_us":897943,"raft.queue_time_us":47,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":180,"replication_time_us":154203,"spinlock_wait_cycles":112512,"thread_start_us":133,"threads_started":1}]]}
W20260804 14:51:28.351281 27335 scanner-internal.cc:458] Time spent opening tablet: real 0.873s user 0.001s sys 0.000s
W20260804 14:51:28.351152 27336 scanner-internal.cc:458] Time spent opening tablet: real 0.872s user 0.001s sys 0.000s
I20260804 14:51:28.658192 27356 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:28.658353 27356 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:28.658891 27356 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:28.659224 26845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:28.659296 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:28.659754 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:28.660110 27356 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:29.792124 27356 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:29.792306 27356 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:29.793046 27356 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:29.793591 26844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:29.793597 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:29.794255 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:29.795492 27375 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50938 (ReqId={client: 2f11fb73142a45a5ab6d1317d6b28d6e, seq_no=3, attempt_no=0}) took 2468 ms. Trace:
I20260804 14:51:29.795589 27375 rpcz_store.cc:276] 0804 14:51:27.326969 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:27.327028 (+ 59us) service_pool.cc:227] Handling call
0804 14:51:29.795476 (+2468448us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:28.194603 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:28.194716 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:28.194722 (+ 6us) write_op.cc:435] Acquired schema lock
0804 14:51:28.194724 (+ 2us) tablet.cc:660] Decoding operations
0804 14:51:28.194814 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:28.194827 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:28.194830 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:28.194878 (+ 48us) tablet.cc:699] Row locks acquired
0804 14:51:28.194880 (+ 2us) write_op.cc:260] PREPARE: finished
0804 14:51:28.194922 (+ 42us) write_op.cc:270] Start()
0804 14:51:28.888104 (+693182us) write_op.cc:276] Timestamp: P: 1785855088194868 usec, L: 0
0804 14:51:28.888108 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:51:28.888361 (+ 253us) log.cc:842] Serialized 354 byte log entry
0804 14:51:29.794310 (+905949us) op_driver.cc:444] REPLICATION: finished
0804 14:51:29.794670 (+ 360us) write_op.cc:301] APPLY: starting
0804 14:51:29.794691 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0804 14:51:29.794734 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0804 14:51:29.794747 (+ 13us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:51:29.794806 (+ 59us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:51:29.794833 (+ 27us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:51:29.794836 (+ 3us) write_op.cc:312] APPLY: finished
0804 14:51:29.794902 (+ 66us) log.cc:842] Serialized 22 byte log entry
0804 14:51:29.795376 (+ 474us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:29.795390 (+ 14us) write_op.cc:454] Released schema lock
0804 14:51:29.795396 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":293,"num_ops":1,"prepare.queue_time_us":867036,"prepare.run_cpu_time_us":736,"prepare.run_wall_time_us":693823,"replication_time_us":906174,"thread_start_us":75,"threads_started":1}]]}
I20260804 14:51:29.796273 26964 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50938 (request call id 9) took 1293 ms. Trace:
I20260804 14:51:29.796332 26967 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50938 (request call id 10) took 1293 ms. Trace:
I20260804 14:51:29.796398 26967 rpcz_store.cc:276] 0804 14:51:28.502843 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:28.502916 (+ 73us) service_pool.cc:227] Handling call
0804 14:51:28.503058 (+ 142us) tablet_service.cc:2780] Created scanner e565eb7ddf6f4f65bb0e6ea607467acd for tablet df07e91451bc49ba978359f97a315ebe, query id is 6c2517b9e73d4fe7aff6937eb8d54525
0804 14:51:28.503258 (+ 200us) tablet_service.cc:2920] Creating iterator
0804 14:51:28.503282 (+ 24us) tablet_service.cc:3297] Waiting safe time to advance
0804 14:51:28.888235 (+384953us) tablet_service.cc:3304] Waiting for operations to commit
0804 14:51:29.795390 (+907155us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1292094 microseconds
0804 14:51:29.795494 (+ 104us) tablet_service.cc:2945] Iterator created
0804 14:51:29.795813 (+ 319us) tablet_service.cc:2967] Iterator init: OK
0804 14:51:29.795841 (+ 28us) tablet_service.cc:3010] has_more: true
0804 14:51:29.795873 (+ 32us) tablet_service.cc:3027] Continuing scan request
0804 14:51:29.795932 (+ 59us) tablet_service.cc:3091] Found scanner e565eb7ddf6f4f65bb0e6ea607467acd for tablet df07e91451bc49ba978359f97a315ebe, query id is 6c2517b9e73d4fe7aff6937eb8d54525
0804 14:51:29.796322 (+ 390us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
I20260804 14:51:29.796361 26964 rpcz_store.cc:276] 0804 14:51:28.502741 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:28.502812 (+ 71us) service_pool.cc:227] Handling call
0804 14:51:28.502978 (+ 166us) tablet_service.cc:2780] Created scanner 1416e5ab76fb486e9605aa12ad6bb829 for tablet df07e91451bc49ba978359f97a315ebe, query id is cc6eeacacb204e43b6d401050536cdc8
0804 14:51:28.503180 (+ 202us) tablet_service.cc:2920] Creating iterator
0804 14:51:28.503237 (+ 57us) tablet_service.cc:3297] Waiting safe time to advance
0804 14:51:28.888875 (+385638us) tablet_service.cc:3304] Waiting for operations to commit
0804 14:51:29.795397 (+906522us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1292147 microseconds
0804 14:51:29.795511 (+ 114us) tablet_service.cc:2945] Iterator created
0804 14:51:29.795804 (+ 293us) tablet_service.cc:2967] Iterator init: OK
0804 14:51:29.795826 (+ 22us) tablet_service.cc:3010] has_more: true
0804 14:51:29.795879 (+ 53us) tablet_service.cc:3027] Continuing scan request
0804 14:51:29.795938 (+ 59us) tablet_service.cc:3091] Found scanner 1416e5ab76fb486e9605aa12ad6bb829 for tablet df07e91451bc49ba978359f97a315ebe, query id is cc6eeacacb204e43b6d401050536cdc8
0804 14:51:29.796265 (+ 327us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":1792}
W20260804 14:51:29.797068 27336 scanner-internal.cc:458] Time spent opening tablet: real 1.295s user 0.000s sys 0.000s
W20260804 14:51:29.797068 27335 scanner-internal.cc:458] Time spent opening tablet: real 1.295s user 0.000s sys 0.000s
I20260804 14:51:29.836110 27363 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:29.836280 27363 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:29.836961 27363 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:29.837976 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:29.838029 27199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:29.838559 26755 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:29.838785 27363 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:30.208343 27356 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:30.209952 27375 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50938 (ReqId={client: 2f11fb73142a45a5ab6d1317d6b28d6e, seq_no=4, attempt_no=0}) took 1858 ms. Trace:
I20260804 14:51:30.210060 27375 rpcz_store.cc:276] 0804 14:51:28.351498 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:28.351541 (+ 43us) service_pool.cc:227] Handling call
0804 14:51:30.209935 (+1858394us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:28.888487 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:28.888635 (+ 148us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:28.888640 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:51:28.888642 (+ 2us) tablet.cc:660] Decoding operations
0804 14:51:28.888728 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:28.888738 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:28.888741 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:28.888797 (+ 56us) tablet.cc:699] Row locks acquired
0804 14:51:28.888798 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:51:28.888838 (+ 40us) write_op.cc:270] Start()
0804 14:51:29.229055 (+340217us) write_op.cc:276] Timestamp: P: 1785855088888806 usec, L: 0
0804 14:51:29.229060 (+ 5us) op_driver.cc:331] REPLICATION: starting
0804 14:51:29.229328 (+ 268us) log.cc:842] Serialized 354 byte log entry
0804 14:51:30.209356 (+980028us) op_driver.cc:444] REPLICATION: finished
0804 14:51:30.209459 (+ 103us) write_op.cc:301] APPLY: starting
0804 14:51:30.209501 (+ 42us) tablet.cc:1365] starting BulkCheckPresence
0804 14:51:30.209536 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0804 14:51:30.209555 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:51:30.209619 (+ 64us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:51:30.209653 (+ 34us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:51:30.209658 (+ 5us) write_op.cc:312] APPLY: finished
0804 14:51:30.209765 (+ 107us) log.cc:842] Serialized 22 byte log entry
0804 14:51:30.209840 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:30.209852 (+ 12us) write_op.cc:454] Released schema lock
0804 14:51:30.209857 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":47,"num_ops":1,"prepare.queue_time_us":536546,"prepare.run_cpu_time_us":825,"prepare.run_wall_time_us":340917,"replication_time_us":980271}]]}
I20260804 14:51:30.689019 27356 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:30.689200 27356 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:30.689738 27356 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:30.690272 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:30.690376 26844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:30.690881 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:30.691121 27356 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260804 14:51:30.692502 27336 scanner-internal.cc:458] Time spent opening tablet: real 0.745s user 0.001s sys 0.000s
W20260804 14:51:30.692878 27335 scanner-internal.cc:458] Time spent opening tablet: real 0.745s user 0.001s sys 0.000s
I20260804 14:51:30.852490 27363 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:30.852679 27363 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:30.853101 26845 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.24.6.130:45113 (request call id 583) took 1013 ms. Trace:
I20260804 14:51:30.853209 27363 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:30.853729 27199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:30.853729 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:30.854336 26757 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:30.853223 26845 rpcz_store.cc:276] 0804 14:51:29.839265 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:29.839338 (+ 73us) service_pool.cc:227] Handling call
0804 14:51:29.839516 (+ 178us) raft_consensus.cc:1457] Updating replica for 2 ops
0804 14:51:29.839611 (+ 95us) raft_consensus.cc:1520] Early marking committed up to index 5
0804 14:51:29.839780 (+ 169us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0804 14:51:29.839894 (+ 114us) write_op.cc:270] Start()
0804 14:51:29.944094 (+104200us) write_op.cc:276] Timestamp: P: 1785855088888806 usec, L: 0
0804 14:51:29.944499 (+ 405us) write_op.cc:270] Start()
0804 14:51:30.851750 (+907251us) write_op.cc:276] Timestamp: P: 1785855089798288 usec, L: 0
0804 14:51:30.852247 (+ 497us) log.cc:842] Serialized 708 byte log entry
0804 14:51:30.852398 (+ 151us) raft_consensus.cc:1631] Marking committed up to 5
0804 14:51:30.852405 (+ 7us) raft_consensus.cc:1683] Filling consensus response to leader.
0804 14:51:30.852437 (+ 32us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0804 14:51:30.853022 (+ 585us) raft_consensus.cc:1671] finished
0804 14:51:30.853024 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0804 14:51:30.853081 (+ 57us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:29.944342 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:29.944504 (+ 162us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:29.944510 (+ 6us) write_op.cc:435] Acquired schema lock
0804 14:51:29.944513 (+ 3us) tablet.cc:660] Decoding operations
0804 14:51:29.944613 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:29.944624 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:29.944627 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:29.944667 (+ 40us) tablet.cc:699] Row locks acquired
0804 14:51:29.944668 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0804 14:51:30.852154 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:30.852259 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:30.852263 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:51:30.852265 (+ 2us) tablet.cc:660] Decoding operations
0804 14:51:30.852339 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:30.852374 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:30.852376 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:30.852492 (+ 116us) tablet.cc:699] Row locks acquired
0804 14:51:30.852494 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":99,"threads_started":1,"wal-append.queue_time_us":195,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":71,"prepare.run_cpu_time_us":413,"prepare.run_wall_time_us":414}],["op",{"num_ops":1,"partition_lock_wait_us":30,"prepare.queue_time_us":238,"prepare.run_cpu_time_us":454,"prepare.run_wall_time_us":454,"thread_start_us":99,"threads_started":1}]]}
I20260804 14:51:30.854540 27363 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:31.007391 27356 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:31.007568 27356 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:31.008132 27356 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:31.008750 26844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:31.008795 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:31.009418 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:31.009917 27356 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:31.145737 27363 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:31.145891 27363 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:31.146423 27363 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:31.146809 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:31.147038 27199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:31.147450 26755 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:31.147693 27363 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:31.639798 27363 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:31.639952 27363 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:31.640551 27363 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:31.640882 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:31.641054 27199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:31.641530 26755 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:31.641732 27363 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260804 14:51:31.643500 27335 scanner-internal.cc:458] Time spent opening tablet: real 0.800s user 0.001s sys 0.000s
W20260804 14:51:31.644263 27336 scanner-internal.cc:458] Time spent opening tablet: real 0.801s user 0.001s sys 0.000s
I20260804 14:51:31.932906 27356 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:31.933063 27356 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:31.933629 27356 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:31.934406 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:31.934291 26845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:31.935101 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:31.935386 27356 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:32.594082 27356 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:32.594269 27356 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:32.594867 27356 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:32.595467 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:32.595768 26844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:32.596231 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:32.597074 27401 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50938 (ReqId={client: 2f11fb73142a45a5ab6d1317d6b28d6e, seq_no=8, attempt_no=0}) took 1586 ms. Trace:
I20260804 14:51:32.597157 27401 rpcz_store.cc:276] 0804 14:51:31.010934 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:31.011011 (+ 77us) service_pool.cc:227] Handling call
0804 14:51:32.597059 (+1586048us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:31.083837 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:31.083975 (+ 138us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:31.083981 (+ 6us) write_op.cc:435] Acquired schema lock
0804 14:51:31.083982 (+ 1us) tablet.cc:660] Decoding operations
0804 14:51:31.084075 (+ 93us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:31.084085 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:31.084088 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:31.084145 (+ 57us) tablet.cc:699] Row locks acquired
0804 14:51:31.084148 (+ 3us) write_op.cc:260] PREPARE: finished
0804 14:51:31.084207 (+ 59us) write_op.cc:270] Start()
0804 14:51:31.997430 (+913223us) write_op.cc:276] Timestamp: P: 1785855091084154 usec, L: 0
0804 14:51:31.997435 (+ 5us) op_driver.cc:331] REPLICATION: starting
0804 14:51:31.997678 (+ 243us) log.cc:842] Serialized 354 byte log entry
0804 14:51:32.595538 (+597860us) op_driver.cc:444] REPLICATION: finished
0804 14:51:32.595823 (+ 285us) write_op.cc:301] APPLY: starting
0804 14:51:32.595844 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0804 14:51:32.595882 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
0804 14:51:32.595895 (+ 13us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:51:32.595964 (+ 69us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:51:32.595990 (+ 26us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:51:32.595994 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:51:32.596090 (+ 96us) log.cc:842] Serialized 22 byte log entry
0804 14:51:32.596969 (+ 879us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:32.596981 (+ 12us) write_op.cc:454] Released schema lock
0804 14:51:32.596986 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":216,"num_ops":1,"prepare.queue_time_us":72380,"prepare.run_cpu_time_us":1028,"prepare.run_wall_time_us":914108,"raft.queue_time_us":238,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":166,"replication_time_us":598079,"thread_start_us":215,"threads_started":2}]]}
W20260804 14:51:32.598122 27335 scanner-internal.cc:458] Time spent opening tablet: real 0.804s user 0.001s sys 0.000s
W20260804 14:51:32.598186 27336 scanner-internal.cc:458] Time spent opening tablet: real 0.803s user 0.001s sys 0.000s
I20260804 14:51:32.688264 27356 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:32.689996 27401 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50938 (ReqId={client: 2f11fb73142a45a5ab6d1317d6b28d6e, seq_no=9, attempt_no=0}) took 1045 ms. Trace:
I20260804 14:51:32.690173 27401 rpcz_store.cc:276] 0804 14:51:31.644456 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:31.644520 (+ 64us) service_pool.cc:227] Handling call
0804 14:51:32.689964 (+1045444us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:31.997977 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:31.998086 (+ 109us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:31.998091 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:51:31.998092 (+ 1us) tablet.cc:660] Decoding operations
0804 14:51:31.998173 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:31.998181 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:31.998183 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:31.998220 (+ 37us) tablet.cc:699] Row locks acquired
0804 14:51:31.998221 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:51:31.998253 (+ 32us) write_op.cc:270] Start()
0804 14:51:32.031457 (+ 33204us) write_op.cc:276] Timestamp: P: 1785855091998197 usec, L: 0
0804 14:51:32.031462 (+ 5us) op_driver.cc:331] REPLICATION: starting
0804 14:51:32.031691 (+ 229us) log.cc:842] Serialized 354 byte log entry
0804 14:51:32.689305 (+657614us) op_driver.cc:444] REPLICATION: finished
0804 14:51:32.689471 (+ 166us) write_op.cc:301] APPLY: starting
0804 14:51:32.689495 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
0804 14:51:32.689542 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0804 14:51:32.689565 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:51:32.689628 (+ 63us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:51:32.689661 (+ 33us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:51:32.689665 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:51:32.689747 (+ 82us) log.cc:842] Serialized 22 byte log entry
0804 14:51:32.689849 (+ 102us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:32.689866 (+ 17us) write_op.cc:454] Released schema lock
0804 14:51:32.689872 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":56,"num_ops":1,"prepare.queue_time_us":353134,"prepare.run_cpu_time_us":813,"prepare.run_wall_time_us":33904,"replication_time_us":657819}]]}
I20260804 14:51:32.830778 27363 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:32.830987 27363 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:32.831553 27363 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:32.831848 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:32.832547 27197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:32.833030 26755 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:33.507259 27363 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:34.039629 27363 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:34.039796 27363 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:34.040328 27363 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:34.040757 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:34.040825 27199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:34.041364 26757 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:34.041604 27363 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:34.042394 27420 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50938 (ReqId={client: 2f11fb73142a45a5ab6d1317d6b28d6e, seq_no=10, attempt_no=0}) took 1443 ms. Trace:
I20260804 14:51:34.042598 26964 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50938 (request call id 24) took 1293 ms. Trace:
I20260804 14:51:34.042694 26964 rpcz_store.cc:276] 0804 14:51:32.749345 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:32.749397 (+ 52us) service_pool.cc:227] Handling call
0804 14:51:32.749599 (+ 202us) tablet_service.cc:2780] Created scanner 2e6efbe9923941e2a5d04da3927cd949 for tablet df07e91451bc49ba978359f97a315ebe, query id is 5e21c3539b7442b59fce2451489b74b9
0804 14:51:32.749849 (+ 250us) tablet_service.cc:2920] Creating iterator
0804 14:51:32.749879 (+ 30us) tablet_service.cc:3297] Waiting safe time to advance
0804 14:51:33.502660 (+752781us) tablet_service.cc:3304] Waiting for operations to commit
0804 14:51:34.041644 (+538984us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1291750 microseconds
0804 14:51:34.041736 (+ 92us) tablet_service.cc:2945] Iterator created
0804 14:51:34.042048 (+ 312us) tablet_service.cc:2967] Iterator init: OK
0804 14:51:34.042075 (+ 27us) tablet_service.cc:3010] has_more: true
0804 14:51:34.042130 (+ 55us) tablet_service.cc:3027] Continuing scan request
0804 14:51:34.042191 (+ 61us) tablet_service.cc:3091] Found scanner 2e6efbe9923941e2a5d04da3927cd949 for tablet df07e91451bc49ba978359f97a315ebe, query id is 5e21c3539b7442b59fce2451489b74b9
0804 14:51:34.042585 (+ 394us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
I20260804 14:51:34.043085 26967 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50938 (request call id 25) took 1293 ms. Trace:
I20260804 14:51:34.042496 27420 rpcz_store.cc:276] 0804 14:51:32.598588 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:32.598658 (+ 70us) service_pool.cc:227] Handling call
0804 14:51:34.042379 (+1443721us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:32.599133 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:32.599249 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:32.599253 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:51:32.599255 (+ 2us) tablet.cc:660] Decoding operations
0804 14:51:32.599326 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:32.599333 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:32.599335 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:32.599366 (+ 31us) tablet.cc:699] Row locks acquired
0804 14:51:32.599367 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:51:32.599395 (+ 28us) write_op.cc:270] Start()
0804 14:51:33.502541 (+903146us) write_op.cc:276] Timestamp: P: 1785855092599334 usec, L: 0
0804 14:51:33.502545 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:51:33.502790 (+ 245us) log.cc:842] Serialized 354 byte log entry
0804 14:51:34.041038 (+538248us) op_driver.cc:444] REPLICATION: finished
0804 14:51:34.041237 (+ 199us) write_op.cc:301] APPLY: starting
0804 14:51:34.041257 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0804 14:51:34.041294 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0804 14:51:34.041310 (+ 16us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:51:34.041370 (+ 60us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:51:34.041403 (+ 33us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:51:34.041408 (+ 5us) write_op.cc:312] APPLY: finished
0804 14:51:34.041477 (+ 69us) log.cc:842] Serialized 22 byte log entry
0804 14:51:34.042282 (+ 805us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:34.042297 (+ 15us) write_op.cc:454] Released schema lock
0804 14:51:34.042302 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":130,"num_ops":1,"prepare.queue_time_us":169,"prepare.run_cpu_time_us":678,"prepare.run_wall_time_us":903749,"raft.queue_time_us":58,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":151,"replication_time_us":538470,"thread_start_us":148,"threads_started":2}]]}
I20260804 14:51:34.043149 26967 rpcz_store.cc:276] 0804 14:51:32.750052 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:32.750121 (+ 69us) service_pool.cc:227] Handling call
0804 14:51:32.750283 (+ 162us) tablet_service.cc:2780] Created scanner 439b39d10c2844edbe0c427a4eab193c for tablet df07e91451bc49ba978359f97a315ebe, query id is 138ce9ae352d4aa49ecacd0fbb240fd4
0804 14:51:32.750495 (+ 212us) tablet_service.cc:2920] Creating iterator
0804 14:51:32.750525 (+ 30us) tablet_service.cc:3297] Waiting safe time to advance
0804 14:51:33.502672 (+752147us) tablet_service.cc:3304] Waiting for operations to commit
0804 14:51:34.042304 (+539632us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1291766 microseconds
0804 14:51:34.042368 (+ 64us) tablet_service.cc:2945] Iterator created
0804 14:51:34.042676 (+ 308us) tablet_service.cc:2967] Iterator init: OK
0804 14:51:34.042701 (+ 25us) tablet_service.cc:3010] has_more: true
0804 14:51:34.042731 (+ 30us) tablet_service.cc:3027] Continuing scan request
0804 14:51:34.042774 (+ 43us) tablet_service.cc:3091] Found scanner 439b39d10c2844edbe0c427a4eab193c for tablet df07e91451bc49ba978359f97a315ebe, query id is 138ce9ae352d4aa49ecacd0fbb240fd4
0804 14:51:34.043077 (+ 303us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
W20260804 14:51:34.043320 27335 scanner-internal.cc:458] Time spent opening tablet: real 1.295s user 0.001s sys 0.000s
W20260804 14:51:34.043766 27336 scanner-internal.cc:458] Time spent opening tablet: real 1.295s user 0.001s sys 0.000s
I20260804 14:51:34.496789 27413 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:34.496989 27413 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:34.497588 27413 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:34.498054 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:34.498090 26844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:34.498608 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:34.515601 27413 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:34.517452 27420 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50938 (ReqId={client: 2f11fb73142a45a5ab6d1317d6b28d6e, seq_no=11, attempt_no=0}) took 1825 ms. Trace:
I20260804 14:51:34.517563 27420 rpcz_store.cc:276] 0804 14:51:32.691587 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:32.691657 (+ 70us) service_pool.cc:227] Handling call
0804 14:51:34.517435 (+1825778us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:33.502978 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:33.503099 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:33.503104 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:51:33.503106 (+ 2us) tablet.cc:660] Decoding operations
0804 14:51:33.503192 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:33.503229 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:33.503236 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:33.503285 (+ 49us) tablet.cc:699] Row locks acquired
0804 14:51:33.503286 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:51:33.503342 (+ 56us) write_op.cc:270] Start()
0804 14:51:33.875599 (+372257us) write_op.cc:276] Timestamp: P: 1785855093503281 usec, L: 0
0804 14:51:33.875606 (+ 7us) op_driver.cc:331] REPLICATION: starting
0804 14:51:33.875857 (+ 251us) log.cc:842] Serialized 354 byte log entry
0804 14:51:34.516756 (+640899us) op_driver.cc:444] REPLICATION: finished
0804 14:51:34.517002 (+ 246us) write_op.cc:301] APPLY: starting
0804 14:51:34.517024 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0804 14:51:34.517061 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0804 14:51:34.517085 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:51:34.517160 (+ 75us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:51:34.517189 (+ 29us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:51:34.517193 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:51:34.517259 (+ 66us) log.cc:842] Serialized 22 byte log entry
0804 14:51:34.517333 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:34.517350 (+ 17us) write_op.cc:454] Released schema lock
0804 14:51:34.517355 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":51,"num_ops":1,"prepare.queue_time_us":810945,"prepare.run_cpu_time_us":874,"prepare.run_wall_time_us":372976,"replication_time_us":641121}]]}
I20260804 14:51:34.613992 27363 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:34.614184 27363 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:34.614765 27363 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:34.615191 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:34.615295 27197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:34.615705 26755 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:34.615899 27363 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:35.240837 27413 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:35.241014 27413 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:35.241595 27413 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:35.242071 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:35.242103 26844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:35.242645 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:35.243635 26964 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50938 (request call id 28) took 1048 ms. Trace:
I20260804 14:51:35.243767 26967 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50938 (request call id 27) took 1049 ms. Trace:
I20260804 14:51:35.243798 27432 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50938 (ReqId={client: 2f11fb73142a45a5ab6d1317d6b28d6e, seq_no=12, attempt_no=0}) took 1199 ms. Trace:
I20260804 14:51:35.243712 26964 rpcz_store.cc:276] 0804 14:51:34.194909 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:34.194978 (+ 69us) service_pool.cc:227] Handling call
0804 14:51:34.195178 (+ 200us) tablet_service.cc:2780] Created scanner 5a7abb6c7d8e4596a57d1be6d8a986ca for tablet df07e91451bc49ba978359f97a315ebe, query id is 20a596619abc494ba45ce06fa6310a14
0804 14:51:34.195427 (+ 249us) tablet_service.cc:2920] Creating iterator
0804 14:51:34.195493 (+ 66us) tablet_service.cc:3297] Waiting safe time to advance
0804 14:51:34.749128 (+553635us) tablet_service.cc:3304] Waiting for operations to commit
0804 14:51:35.242669 (+493541us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1047164 microseconds
0804 14:51:35.242762 (+ 93us) tablet_service.cc:2945] Iterator created
0804 14:51:35.243131 (+ 369us) tablet_service.cc:2967] Iterator init: OK
0804 14:51:35.243156 (+ 25us) tablet_service.cc:3010] has_more: true
0804 14:51:35.243182 (+ 26us) tablet_service.cc:3027] Continuing scan request
0804 14:51:35.243277 (+ 95us) tablet_service.cc:3091] Found scanner 5a7abb6c7d8e4596a57d1be6d8a986ca for tablet df07e91451bc49ba978359f97a315ebe, query id is 20a596619abc494ba45ce06fa6310a14
0804 14:51:35.243620 (+ 343us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":56960}
I20260804 14:51:35.243867 26967 rpcz_store.cc:276] 0804 14:51:34.194837 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:34.194971 (+ 134us) service_pool.cc:227] Handling call
0804 14:51:34.195246 (+ 275us) tablet_service.cc:2780] Created scanner 794b1f64ade3427d8f28ee9a709922fe for tablet df07e91451bc49ba978359f97a315ebe, query id is 7bfd945b7d0a473bb763780ef159c5b4
0804 14:51:34.195431 (+ 185us) tablet_service.cc:2920] Creating iterator
0804 14:51:34.195493 (+ 62us) tablet_service.cc:3297] Waiting safe time to advance
0804 14:51:34.748502 (+553009us) tablet_service.cc:3304] Waiting for operations to commit
0804 14:51:35.242885 (+494383us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1047377 microseconds
0804 14:51:35.242973 (+ 88us) tablet_service.cc:2945] Iterator created
0804 14:51:35.243277 (+ 304us) tablet_service.cc:2967] Iterator init: OK
0804 14:51:35.243303 (+ 26us) tablet_service.cc:3010] has_more: true
0804 14:51:35.243330 (+ 27us) tablet_service.cc:3027] Continuing scan request
0804 14:51:35.243408 (+ 78us) tablet_service.cc:3091] Found scanner 794b1f64ade3427d8f28ee9a709922fe for tablet df07e91451bc49ba978359f97a315ebe, query id is 7bfd945b7d0a473bb763780ef159c5b4
0804 14:51:35.243756 (+ 348us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":103424}
I20260804 14:51:35.243901 27432 rpcz_store.cc:276] 0804 14:51:34.044455 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:34.044533 (+ 78us) service_pool.cc:227] Handling call
0804 14:51:35.243787 (+1199254us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:34.045034 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:34.045102 (+ 68us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:34.045105 (+ 3us) write_op.cc:435] Acquired schema lock
0804 14:51:34.045106 (+ 1us) tablet.cc:660] Decoding operations
0804 14:51:34.045165 (+ 59us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:34.045172 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:34.045174 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:34.045203 (+ 29us) tablet.cc:699] Row locks acquired
0804 14:51:34.045204 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:51:34.045229 (+ 25us) write_op.cc:270] Start()
0804 14:51:34.748365 (+703136us) write_op.cc:276] Timestamp: P: 1785855094045175 usec, L: 0
0804 14:51:34.748368 (+ 3us) op_driver.cc:331] REPLICATION: starting
0804 14:51:34.748631 (+ 263us) log.cc:842] Serialized 354 byte log entry
0804 14:51:35.242030 (+493399us) op_driver.cc:444] REPLICATION: finished
0804 14:51:35.242297 (+ 267us) write_op.cc:301] APPLY: starting
0804 14:51:35.242317 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0804 14:51:35.242351 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0804 14:51:35.242365 (+ 14us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:51:35.242430 (+ 65us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:51:35.242464 (+ 34us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:51:35.242469 (+ 5us) write_op.cc:312] APPLY: finished
0804 14:51:35.242542 (+ 73us) log.cc:842] Serialized 22 byte log entry
0804 14:51:35.243695 (+ 1153us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:35.243709 (+ 14us) write_op.cc:454] Released schema lock
0804 14:51:35.243715 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":209,"num_ops":1,"prepare.queue_time_us":84,"prepare.run_cpu_time_us":604,"prepare.run_wall_time_us":703665,"replication_time_us":493637,"thread_start_us":100,"threads_started":1}]]}
W20260804 14:51:35.244469 27336 scanner-internal.cc:458] Time spent opening tablet: real 1.050s user 0.000s sys 0.000s
W20260804 14:51:35.244638 27335 scanner-internal.cc:458] Time spent opening tablet: real 1.051s user 0.001s sys 0.000s
I20260804 14:51:35.637032 27363 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:35.637220 27363 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:35.637894 27363 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:35.638314 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:35.638440 27199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:35.638964 26757 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:35.639389 27363 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:35.724961 27413 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:35.726768 27432 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50938 (ReqId={client: 2f11fb73142a45a5ab6d1317d6b28d6e, seq_no=13, attempt_no=0}) took 1207 ms. Trace:
I20260804 14:51:35.726871 27432 rpcz_store.cc:276] 0804 14:51:34.519252 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:34.519299 (+ 47us) service_pool.cc:227] Handling call
0804 14:51:35.726750 (+1207451us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:34.748735 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:34.748850 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:34.748856 (+ 6us) write_op.cc:435] Acquired schema lock
0804 14:51:34.748858 (+ 2us) tablet.cc:660] Decoding operations
0804 14:51:34.748954 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:34.748990 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:34.748994 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:34.749044 (+ 50us) tablet.cc:699] Row locks acquired
0804 14:51:34.749045 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:51:34.749092 (+ 47us) write_op.cc:270] Start()
0804 14:51:34.821283 (+ 72191us) write_op.cc:276] Timestamp: P: 1785855094749019 usec, L: 0
0804 14:51:34.821287 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:51:34.821529 (+ 242us) log.cc:842] Serialized 354 byte log entry
0804 14:51:35.726144 (+904615us) op_driver.cc:444] REPLICATION: finished
0804 14:51:35.726291 (+ 147us) write_op.cc:301] APPLY: starting
0804 14:51:35.726310 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0804 14:51:35.726341 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
0804 14:51:35.726360 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:51:35.726419 (+ 59us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:51:35.726469 (+ 50us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:51:35.726473 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:51:35.726557 (+ 84us) log.cc:842] Serialized 22 byte log entry
0804 14:51:35.726649 (+ 92us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:35.726663 (+ 14us) write_op.cc:454] Released schema lock
0804 14:51:35.726669 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":58,"num_ops":1,"prepare.queue_time_us":228972,"prepare.run_cpu_time_us":849,"prepare.run_wall_time_us":72926,"replication_time_us":904826}]]}
I20260804 14:51:36.243973 27413 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:36.244204 27413 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:36.244699 27413 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:36.245210 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:36.245215 26844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:36.245748 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:36.245956 27413 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:36.247043 27445 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50938 (ReqId={client: 2f11fb73142a45a5ab6d1317d6b28d6e, seq_no=14, attempt_no=0}) took 1001 ms. Trace:
W20260804 14:51:36.247486 27335 scanner-internal.cc:458] Time spent opening tablet: real 0.852s user 0.001s sys 0.000s
W20260804 14:51:36.247486 27336 scanner-internal.cc:458] Time spent opening tablet: real 0.852s user 0.000s sys 0.000s
I20260804 14:51:36.247328 27445 rpcz_store.cc:276] 0804 14:51:35.245411 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:35.245539 (+ 128us) service_pool.cc:227] Handling call
0804 14:51:36.247028 (+1001489us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:35.245974 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:35.246055 (+ 81us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:35.246058 (+ 3us) write_op.cc:435] Acquired schema lock
0804 14:51:35.246059 (+ 1us) tablet.cc:660] Decoding operations
0804 14:51:35.246120 (+ 61us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:35.246127 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:35.246129 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:35.246160 (+ 31us) tablet.cc:699] Row locks acquired
0804 14:51:35.246161 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:51:35.246189 (+ 28us) write_op.cc:270] Start()
0804 14:51:35.488340 (+242151us) write_op.cc:276] Timestamp: P: 1785855095246121 usec, L: 0
0804 14:51:35.488344 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:51:35.488697 (+ 353us) log.cc:842] Serialized 354 byte log entry
0804 14:51:36.245208 (+756511us) op_driver.cc:444] REPLICATION: finished
0804 14:51:36.245608 (+ 400us) write_op.cc:301] APPLY: starting
0804 14:51:36.245630 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0804 14:51:36.245662 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0804 14:51:36.245677 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:51:36.245862 (+ 185us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:51:36.245903 (+ 41us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:51:36.245909 (+ 6us) write_op.cc:312] APPLY: finished
0804 14:51:36.245980 (+ 71us) log.cc:842] Serialized 22 byte log entry
0804 14:51:36.246913 (+ 933us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:36.246924 (+ 11us) write_op.cc:454] Released schema lock
0804 14:51:36.246928 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":342,"num_ops":1,"prepare.queue_time_us":49,"prepare.run_cpu_time_us":749,"prepare.run_wall_time_us":242806,"replication_time_us":756842,"thread_start_us":71,"threads_started":1}]]}
I20260804 14:51:36.399482 27363 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:36.399632 27363 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:36.400167 27363 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:36.400573 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:36.400765 27197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:36.401185 26755 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:36.401459 27363 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:36.556473 27413 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:36.556660 27413 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:36.557145 27413 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:36.557596 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:36.557603 26845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:36.558158 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:36.560660 27413 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:36.875926 27413 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:36.876112 27413 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:36.876721 27413 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:36.877228 26844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:36.877269 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:36.877767 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:36.878069 27454 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50938 (ReqId={client: 2f11fb73142a45a5ab6d1317d6b28d6e, seq_no=15, attempt_no=0}) took 1149 ms. Trace:
I20260804 14:51:36.878221 27413 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:36.878170 27454 rpcz_store.cc:276] 0804 14:51:35.728649 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:35.728736 (+ 87us) service_pool.cc:227] Handling call
0804 14:51:36.878054 (+1149318us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:35.729256 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:35.729345 (+ 89us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:35.729349 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:51:35.729350 (+ 1us) tablet.cc:660] Decoding operations
0804 14:51:35.729456 (+ 106us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:35.729464 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:35.729466 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:35.729503 (+ 37us) tablet.cc:699] Row locks acquired
0804 14:51:35.729504 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:51:35.729537 (+ 33us) write_op.cc:270] Start()
0804 14:51:36.646711 (+917174us) write_op.cc:276] Timestamp: P: 1785855095729466 usec, L: 0
0804 14:51:36.646715 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:51:36.646989 (+ 274us) log.cc:842] Serialized 354 byte log entry
0804 14:51:36.877117 (+230128us) op_driver.cc:444] REPLICATION: finished
0804 14:51:36.877617 (+ 500us) write_op.cc:301] APPLY: starting
0804 14:51:36.877636 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0804 14:51:36.877673 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0804 14:51:36.877685 (+ 12us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:51:36.877749 (+ 64us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:51:36.877781 (+ 32us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:51:36.877790 (+ 9us) write_op.cc:312] APPLY: finished
0804 14:51:36.877864 (+ 74us) log.cc:842] Serialized 22 byte log entry
0804 14:51:36.877949 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:36.877964 (+ 15us) write_op.cc:454] Released schema lock
0804 14:51:36.877970 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":439,"mutex_wait_us":46,"num_ops":1,"prepare.queue_time_us":60,"prepare.run_cpu_time_us":764,"prepare.run_wall_time_us":917878,"raft.queue_time_us":354,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":214,"replication_time_us":230376,"thread_start_us":111,"threads_started":1}]]}
I20260804 14:51:37.157729 27363 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:37.157919 27363 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:37.158465 27363 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:37.158833 27197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:37.159301 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:37.160190 26757 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:37.160403 27363 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:37.504989 27363 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:37.505194 27363 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:37.505848 27363 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:37.506349 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:37.506349 27199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:37.506810 26755 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:37.507186 27458 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50938 (ReqId={client: 2f11fb73142a45a5ab6d1317d6b28d6e, seq_no=16, attempt_no=0}) took 1258 ms. Trace:
I20260804 14:51:37.507334 27458 rpcz_store.cc:276] 0804 14:51:36.248716 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:36.248780 (+ 64us) service_pool.cc:227] Handling call
0804 14:51:37.507171 (+1258391us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:36.647154 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:36.647348 (+ 194us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:36.647353 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:51:36.647355 (+ 2us) tablet.cc:660] Decoding operations
0804 14:51:36.647437 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:36.647449 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:36.647452 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:36.647497 (+ 45us) tablet.cc:699] Row locks acquired
0804 14:51:36.647498 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:51:36.647542 (+ 44us) write_op.cc:270] Start()
0804 14:51:37.166699 (+519157us) write_op.cc:276] Timestamp: P: 1785855096647488 usec, L: 0
0804 14:51:37.166702 (+ 3us) op_driver.cc:331] REPLICATION: starting
0804 14:51:37.166980 (+ 278us) log.cc:842] Serialized 354 byte log entry
0804 14:51:37.506086 (+339106us) op_driver.cc:444] REPLICATION: finished
0804 14:51:37.506367 (+ 281us) write_op.cc:301] APPLY: starting
0804 14:51:37.506420 (+ 53us) tablet.cc:1365] starting BulkCheckPresence
0804 14:51:37.506457 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0804 14:51:37.506471 (+ 14us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:51:37.506934 (+ 463us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:51:37.506959 (+ 25us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:51:37.506963 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:51:37.507029 (+ 66us) log.cc:842] Serialized 22 byte log entry
0804 14:51:37.507098 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:37.507108 (+ 10us) write_op.cc:454] Released schema lock
0804 14:51:37.507111 (+ 3us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":225,"num_ops":1,"prepare.queue_time_us":398040,"prepare.run_cpu_time_us":818,"prepare.run_wall_time_us":519923,"raft.queue_time_us":45,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":368,"replication_time_us":339357,"spinlock_wait_cycles":19328,"thread_start_us":106,"threads_started":1}]]}
I20260804 14:51:37.508071 27363 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:37.775960 27413 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:37.776142 27413 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:37.776651 27413 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:37.777091 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:37.777110 26844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:37.777611 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:38.390405 27413 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:38.392467 27463 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50938 (ReqId={client: 2f11fb73142a45a5ab6d1317d6b28d6e, seq_no=17, attempt_no=0}) took 1512 ms. Trace:
I20260804 14:51:38.392715 27463 rpcz_store.cc:276] 0804 14:51:36.880121 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:36.880177 (+ 56us) service_pool.cc:227] Handling call
0804 14:51:38.392446 (+1512269us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:37.167119 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:37.167274 (+ 155us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:37.167280 (+ 6us) write_op.cc:435] Acquired schema lock
0804 14:51:37.167282 (+ 2us) tablet.cc:660] Decoding operations
0804 14:51:37.167371 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:37.167382 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:37.167385 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:37.167434 (+ 49us) tablet.cc:699] Row locks acquired
0804 14:51:37.167436 (+ 2us) write_op.cc:260] PREPARE: finished
0804 14:51:37.167481 (+ 45us) write_op.cc:270] Start()
0804 14:51:37.598681 (+431200us) write_op.cc:276] Timestamp: P: 1785855097167413 usec, L: 0
0804 14:51:37.598685 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:51:37.598946 (+ 261us) log.cc:842] Serialized 354 byte log entry
0804 14:51:38.391701 (+792755us) op_driver.cc:444] REPLICATION: finished
0804 14:51:38.392024 (+ 323us) write_op.cc:301] APPLY: starting
0804 14:51:38.392043 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0804 14:51:38.392086 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0804 14:51:38.392104 (+ 18us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:51:38.392174 (+ 70us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:51:38.392200 (+ 26us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:51:38.392204 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:51:38.392265 (+ 61us) log.cc:842] Serialized 22 byte log entry
0804 14:51:38.392341 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:38.392354 (+ 13us) write_op.cc:454] Released schema lock
0804 14:51:38.392358 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":229,"num_ops":1,"prepare.queue_time_us":286593,"prepare.run_cpu_time_us":860,"prepare.run_wall_time_us":431963,"replication_time_us":792994,"thread_start_us":86,"threads_started":1}]]}
W20260804 14:51:38.394250 27335 scanner-internal.cc:458] Time spent opening tablet: real 0.735s user 0.001s sys 0.000s
W20260804 14:51:38.394361 27336 scanner-internal.cc:458] Time spent opening tablet: real 0.735s user 0.001s sys 0.000s
I20260804 14:51:38.431887 27363 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:38.432052 27363 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:38.432644 27363 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:38.433131 27197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:38.433533 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:38.433972 26757 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:38.546021 27363 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:39.031890 27363 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:39.032064 27363 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:39.032735 27363 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:39.033205 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:39.033264 27199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:39.033869 26757 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:39.318048 27363 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:39.775245 27471 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:39.775425 27471 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:39.775974 27471 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:39.776405 26845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:39.776768 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:39.777456 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:39.777661 27471 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:40.340157 27482 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:40.340343 27482 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:40.340970 27482 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:40.341427 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:40.341470 27199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:40.342145 26757 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:40.342330 27482 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:40.343438 26964 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50938 (request call id 49) took 1155 ms. Trace:
I20260804 14:51:40.343510 26967 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50938 (request call id 48) took 1156 ms. Trace:
I20260804 14:51:40.343607 26967 rpcz_store.cc:276] 0804 14:51:39.187326 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:39.187417 (+ 91us) service_pool.cc:227] Handling call
0804 14:51:39.187647 (+ 230us) tablet_service.cc:2780] Created scanner 909310bb713a4d79bce5164d17f4dc0f for tablet df07e91451bc49ba978359f97a315ebe, query id is d4abee89dec54e6fa31adf98a9744445
0804 14:51:39.187885 (+ 238us) tablet_service.cc:2920] Creating iterator
0804 14:51:39.187965 (+ 80us) tablet_service.cc:3297] Waiting safe time to advance
0804 14:51:39.866938 (+678973us) tablet_service.cc:3304] Waiting for operations to commit
0804 14:51:40.342533 (+475595us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1154552 microseconds
0804 14:51:40.342647 (+ 114us) tablet_service.cc:2945] Iterator created
0804 14:51:40.342984 (+ 337us) tablet_service.cc:2967] Iterator init: OK
0804 14:51:40.343009 (+ 25us) tablet_service.cc:3010] has_more: true
0804 14:51:40.343035 (+ 26us) tablet_service.cc:3027] Continuing scan request
0804 14:51:40.343081 (+ 46us) tablet_service.cc:3091] Found scanner 909310bb713a4d79bce5164d17f4dc0f for tablet df07e91451bc49ba978359f97a315ebe, query id is d4abee89dec54e6fa31adf98a9744445
0804 14:51:40.343496 (+ 415us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":17664}
I20260804 14:51:40.343534 26964 rpcz_store.cc:276] 0804 14:51:39.187481 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:39.187549 (+ 68us) service_pool.cc:227] Handling call
0804 14:51:39.187647 (+ 98us) tablet_service.cc:2780] Created scanner ef2ef3a34a3d4c738e731123359536bc for tablet df07e91451bc49ba978359f97a315ebe, query id is 531ac9be51294d13b919ddaddb56b8d4
0804 14:51:39.187885 (+ 238us) tablet_service.cc:2920] Creating iterator
0804 14:51:39.187966 (+ 81us) tablet_service.cc:3297] Waiting safe time to advance
0804 14:51:39.866946 (+678980us) tablet_service.cc:3304] Waiting for operations to commit
0804 14:51:40.342550 (+475604us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1154567 microseconds
0804 14:51:40.342643 (+ 93us) tablet_service.cc:2945] Iterator created
0804 14:51:40.342968 (+ 325us) tablet_service.cc:2967] Iterator init: OK
0804 14:51:40.342993 (+ 25us) tablet_service.cc:3010] has_more: true
0804 14:51:40.343026 (+ 33us) tablet_service.cc:3027] Continuing scan request
0804 14:51:40.343080 (+ 54us) tablet_service.cc:3091] Found scanner ef2ef3a34a3d4c738e731123359536bc for tablet df07e91451bc49ba978359f97a315ebe, query id is 531ac9be51294d13b919ddaddb56b8d4
0804 14:51:40.343427 (+ 347us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":127360}
I20260804 14:51:40.344043 27490 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50938 (ReqId={client: 2f11fb73142a45a5ab6d1317d6b28d6e, seq_no=21, attempt_no=0}) took 1308 ms. Trace:
W20260804 14:51:40.344339 27335 scanner-internal.cc:458] Time spent opening tablet: real 1.158s user 0.001s sys 0.000s
I20260804 14:51:40.344139 27490 rpcz_store.cc:276] 0804 14:51:39.035873 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:39.035947 (+ 74us) service_pool.cc:227] Handling call
0804 14:51:40.344029 (+1308082us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:39.036361 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:39.036454 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:39.036458 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:51:39.036460 (+ 2us) tablet.cc:660] Decoding operations
0804 14:51:39.036519 (+ 59us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:39.036526 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:39.036528 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:39.036559 (+ 31us) tablet.cc:699] Row locks acquired
0804 14:51:39.036560 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:51:39.036588 (+ 28us) write_op.cc:270] Start()
0804 14:51:39.866757 (+830169us) write_op.cc:276] Timestamp: P: 1785855099036549 usec, L: 0
0804 14:51:39.866761 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:51:39.867063 (+ 302us) log.cc:842] Serialized 354 byte log entry
0804 14:51:40.341537 (+474474us) op_driver.cc:444] REPLICATION: finished
0804 14:51:40.341927 (+ 390us) write_op.cc:301] APPLY: starting
0804 14:51:40.341950 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0804 14:51:40.341987 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0804 14:51:40.342003 (+ 16us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:51:40.342079 (+ 76us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:51:40.342107 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:51:40.342111 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:51:40.342176 (+ 65us) log.cc:842] Serialized 22 byte log entry
0804 14:51:40.343927 (+ 1751us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:40.343941 (+ 14us) write_op.cc:454] Released schema lock
0804 14:51:40.343946 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":299,"num_ops":1,"prepare.queue_time_us":51,"prepare.run_cpu_time_us":737,"prepare.run_wall_time_us":830823,"raft.queue_time_us":62,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":159,"replication_time_us":474750,"spinlock_wait_cycles":57600,"thread_start_us":183,"threads_started":1}]]}
W20260804 14:51:40.344339 27336 scanner-internal.cc:458] Time spent opening tablet: real 1.158s user 0.001s sys 0.000s
I20260804 14:51:40.474160 27471 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:40.474330 27471 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:40.474831 27471 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:40.475164 26844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:40.475431 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:40.475847 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:40.877031 27471 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:41.166774 27482 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:41.166944 27482 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:41.167517 27482 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:41.168133 27199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:41.168432 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:41.169067 26757 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:41.169348 27499 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50938 (ReqId={client: 2f11fb73142a45a5ab6d1317d6b28d6e, seq_no=22, attempt_no=0}) took 1848 ms. Trace:
I20260804 14:51:41.169603 27482 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:41.169417 27499 rpcz_store.cc:276] 0804 14:51:39.320778 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:39.320848 (+ 70us) service_pool.cc:227] Handling call
0804 14:51:41.169334 (+1848486us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:39.867263 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:39.867381 (+ 118us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:39.867386 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:51:39.867388 (+ 2us) tablet.cc:660] Decoding operations
0804 14:51:39.867526 (+ 138us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:39.867537 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:39.867541 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:39.867592 (+ 51us) tablet.cc:699] Row locks acquired
0804 14:51:39.867594 (+ 2us) write_op.cc:260] PREPARE: finished
0804 14:51:39.867641 (+ 47us) write_op.cc:270] Start()
0804 14:51:40.719782 (+852141us) write_op.cc:276] Timestamp: P: 1785855099867575 usec, L: 0
0804 14:51:40.719785 (+ 3us) op_driver.cc:331] REPLICATION: starting
0804 14:51:40.720092 (+ 307us) log.cc:842] Serialized 354 byte log entry
0804 14:51:41.167787 (+447695us) op_driver.cc:444] REPLICATION: finished
0804 14:51:41.168093 (+ 306us) write_op.cc:301] APPLY: starting
0804 14:51:41.168113 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0804 14:51:41.168151 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
0804 14:51:41.168172 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:51:41.168237 (+ 65us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:51:41.168271 (+ 34us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:51:41.168275 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:51:41.168346 (+ 71us) log.cc:842] Serialized 22 byte log entry
0804 14:51:41.169233 (+ 887us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:41.169249 (+ 16us) write_op.cc:454] Released schema lock
0804 14:51:41.169255 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":248,"num_ops":1,"prepare.queue_time_us":546017,"prepare.run_cpu_time_us":868,"prepare.run_wall_time_us":852936,"raft.queue_time_us":45,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":131,"replication_time_us":447963,"spinlock_wait_cycles":16256,"thread_start_us":106,"threads_started":1}]]}
W20260804 14:51:41.169994 27336 scanner-internal.cc:458] Time spent opening tablet: real 0.675s user 0.000s sys 0.001s
W20260804 14:51:41.170868 27335 scanner-internal.cc:458] Time spent opening tablet: real 0.676s user 0.001s sys 0.000s
I20260804 14:51:41.590662 27471 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:41.590822 27471 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:41.591418 27471 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:41.591890 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:41.591954 26844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:41.592455 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:41.592670 27471 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:42.342895 27482 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:42.343184 27482 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:42.343935 27482 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:42.344380 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:42.344470 27199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:42.344972 26755 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:42.345846 27508 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50938 (ReqId={client: 2f11fb73142a45a5ab6d1317d6b28d6e, seq_no=23, attempt_no=0}) took 2000 ms. Trace:
I20260804 14:51:42.346058 27508 rpcz_store.cc:276] 0804 14:51:40.345636 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:40.345682 (+ 46us) service_pool.cc:227] Handling call
0804 14:51:42.345671 (+1999989us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:40.720235 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:40.720352 (+ 117us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:40.720359 (+ 7us) write_op.cc:435] Acquired schema lock
0804 14:51:40.720363 (+ 4us) tablet.cc:660] Decoding operations
0804 14:51:40.720453 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:40.720465 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:40.720472 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:40.720521 (+ 49us) tablet.cc:699] Row locks acquired
0804 14:51:40.720523 (+ 2us) write_op.cc:260] PREPARE: finished
0804 14:51:40.720571 (+ 48us) write_op.cc:270] Start()
0804 14:51:41.348754 (+628183us) write_op.cc:276] Timestamp: P: 1785855100720505 usec, L: 0
0804 14:51:41.348757 (+ 3us) op_driver.cc:331] REPLICATION: starting
0804 14:51:41.349066 (+ 309us) log.cc:842] Serialized 354 byte log entry
0804 14:51:42.344463 (+995397us) op_driver.cc:444] REPLICATION: finished
0804 14:51:42.345011 (+ 548us) write_op.cc:301] APPLY: starting
0804 14:51:42.345033 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0804 14:51:42.345069 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0804 14:51:42.345085 (+ 16us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:51:42.345151 (+ 66us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:51:42.345186 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:51:42.345191 (+ 5us) write_op.cc:312] APPLY: finished
0804 14:51:42.345266 (+ 75us) log.cc:842] Serialized 22 byte log entry
0804 14:51:42.345561 (+ 295us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:42.345579 (+ 18us) write_op.cc:454] Released schema lock
0804 14:51:42.345586 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":487,"num_ops":1,"prepare.queue_time_us":374221,"prepare.run_cpu_time_us":844,"prepare.run_wall_time_us":628939,"raft.queue_time_us":42,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":151,"replication_time_us":995678,"thread_start_us":73,"threads_started":1}]]}
I20260804 14:51:42.346529 26964 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50938 (request call id 55) took 1025 ms. Trace:
I20260804 14:51:42.346561 26967 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50938 (request call id 56) took 1024 ms. Trace:
I20260804 14:51:42.346626 26967 rpcz_store.cc:276] 0804 14:51:41.322059 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:41.322125 (+ 66us) service_pool.cc:227] Handling call
0804 14:51:41.322243 (+ 118us) tablet_service.cc:2780] Created scanner 2ad19cc5bac74d9c97b9f090e8cbb71c for tablet df07e91451bc49ba978359f97a315ebe, query id is 84edbd4f418140a1a57e0661d112d7ed
0804 14:51:41.322397 (+ 154us) tablet_service.cc:2920] Creating iterator
0804 14:51:41.322422 (+ 25us) tablet_service.cc:3297] Waiting safe time to advance
0804 14:51:41.348956 (+ 26534us) tablet_service.cc:3304] Waiting for operations to commit
0804 14:51:42.345604 (+996648us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1023171 microseconds
0804 14:51:42.345682 (+ 78us) tablet_service.cc:2945] Iterator created
0804 14:51:42.346059 (+ 377us) tablet_service.cc:2967] Iterator init: OK
0804 14:51:42.346088 (+ 29us) tablet_service.cc:3010] has_more: true
0804 14:51:42.346122 (+ 34us) tablet_service.cc:3027] Continuing scan request
0804 14:51:42.346172 (+ 50us) tablet_service.cc:3091] Found scanner 2ad19cc5bac74d9c97b9f090e8cbb71c for tablet df07e91451bc49ba978359f97a315ebe, query id is 84edbd4f418140a1a57e0661d112d7ed
0804 14:51:42.346548 (+ 376us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
I20260804 14:51:42.346596 26964 rpcz_store.cc:276] 0804 14:51:41.321411 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:41.321484 (+ 73us) service_pool.cc:227] Handling call
0804 14:51:41.321657 (+ 173us) tablet_service.cc:2780] Created scanner c16429091bf44814b94628b5806fe99a for tablet df07e91451bc49ba978359f97a315ebe, query id is 9a44319e7f4449fb8f440b8e410510a0
0804 14:51:41.321905 (+ 248us) tablet_service.cc:2920] Creating iterator
0804 14:51:41.321932 (+ 27us) tablet_service.cc:3297] Waiting safe time to advance
0804 14:51:41.348918 (+ 26986us) tablet_service.cc:3304] Waiting for operations to commit
0804 14:51:42.345583 (+996665us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1023636 microseconds
0804 14:51:42.345681 (+ 98us) tablet_service.cc:2945] Iterator created
0804 14:51:42.346064 (+ 383us) tablet_service.cc:2967] Iterator init: OK
0804 14:51:42.346102 (+ 38us) tablet_service.cc:3010] has_more: true
0804 14:51:42.346126 (+ 24us) tablet_service.cc:3027] Continuing scan request
0804 14:51:42.346168 (+ 42us) tablet_service.cc:3091] Found scanner c16429091bf44814b94628b5806fe99a for tablet df07e91451bc49ba978359f97a315ebe, query id is 9a44319e7f4449fb8f440b8e410510a0
0804 14:51:42.346519 (+ 351us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":17664}
W20260804 14:51:42.347307 27335 scanner-internal.cc:458] Time spent opening tablet: real 1.026s user 0.001s sys 0.000s
W20260804 14:51:42.347318 27336 scanner-internal.cc:458] Time spent opening tablet: real 1.027s user 0.000s sys 0.001s
I20260804 14:51:42.390057 27471 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:42.390226 27471 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:42.390971 27471 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:42.391616 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:42.391629 26844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:42.392160 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:42.510282 27482 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:42.511907 27508 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50938 (ReqId={client: 2f11fb73142a45a5ab6d1317d6b28d6e, seq_no=24, attempt_no=0}) took 1340 ms. Trace:
I20260804 14:51:42.512012 27508 rpcz_store.cc:276] 0804 14:51:41.171380 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:41.171432 (+ 52us) service_pool.cc:227] Handling call
0804 14:51:42.511890 (+1340458us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:41.349203 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:41.349342 (+ 139us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:41.349346 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:51:41.349348 (+ 2us) tablet.cc:660] Decoding operations
0804 14:51:41.349449 (+ 101us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:41.349458 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:41.349460 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:41.349523 (+ 63us) tablet.cc:699] Row locks acquired
0804 14:51:41.349526 (+ 3us) write_op.cc:260] PREPARE: finished
0804 14:51:41.349567 (+ 41us) write_op.cc:270] Start()
0804 14:51:42.046771 (+697204us) write_op.cc:276] Timestamp: P: 1785855101349504 usec, L: 0
0804 14:51:42.046776 (+ 5us) op_driver.cc:331] REPLICATION: starting
0804 14:51:42.047030 (+ 254us) log.cc:842] Serialized 354 byte log entry
0804 14:51:42.511294 (+464264us) op_driver.cc:444] REPLICATION: finished
0804 14:51:42.511402 (+ 108us) write_op.cc:301] APPLY: starting
0804 14:51:42.511423 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0804 14:51:42.511475 (+ 52us) tablet.cc:1368] finished BulkCheckPresence
0804 14:51:42.511495 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:51:42.511594 (+ 99us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:51:42.511630 (+ 36us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:51:42.511635 (+ 5us) write_op.cc:312] APPLY: finished
0804 14:51:42.511714 (+ 79us) log.cc:842] Serialized 22 byte log entry
0804 14:51:42.511770 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:42.511803 (+ 33us) write_op.cc:454] Released schema lock
0804 14:51:42.511810 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":52,"num_ops":1,"prepare.queue_time_us":177465,"prepare.run_cpu_time_us":842,"prepare.run_wall_time_us":697926,"replication_time_us":464493}]]}
I20260804 14:51:42.976401 27471 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:43.021669 27482 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:43.021845 27482 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:43.022503 27482 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:43.022902 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:43.023087 27199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:43.023582 26755 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:43.023833 27482 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260804 14:51:43.025390 27336 scanner-internal.cc:458] Time spent opening tablet: real 0.527s user 0.000s sys 0.001s
W20260804 14:51:43.025408 27335 scanner-internal.cc:458] Time spent opening tablet: real 0.527s user 0.001s sys 0.000s
I20260804 14:51:43.333966 27471 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:43.334133 27471 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:43.334720 27471 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:43.335263 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:43.335327 26844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:43.335817 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:43.969468 27482 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:43.969640 27482 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:43.970214 27482 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:43.970650 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:43.970652 27199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:43.971180 26755 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:43.971534 27482 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:43.972065 27528 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50938 (ReqId={client: 2f11fb73142a45a5ab6d1317d6b28d6e, seq_no=26, attempt_no=0}) took 1458 ms. Trace:
I20260804 14:51:43.972129 27528 rpcz_store.cc:276] 0804 14:51:42.513477 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:42.513550 (+ 73us) service_pool.cc:227] Handling call
0804 14:51:43.972049 (+1458499us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:42.725928 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:42.726072 (+ 144us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:42.726078 (+ 6us) write_op.cc:435] Acquired schema lock
0804 14:51:42.726079 (+ 1us) tablet.cc:660] Decoding operations
0804 14:51:42.726170 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:42.726181 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:42.726184 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:42.726255 (+ 71us) tablet.cc:699] Row locks acquired
0804 14:51:42.726256 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:51:42.726300 (+ 44us) write_op.cc:270] Start()
0804 14:51:43.278463 (+552163us) write_op.cc:276] Timestamp: P: 1785855102726241 usec, L: 0
0804 14:51:43.278466 (+ 3us) op_driver.cc:331] REPLICATION: starting
0804 14:51:43.278682 (+ 216us) log.cc:842] Serialized 354 byte log entry
0804 14:51:43.971281 (+692599us) op_driver.cc:444] REPLICATION: finished
0804 14:51:43.971587 (+ 306us) write_op.cc:301] APPLY: starting
0804 14:51:43.971610 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0804 14:51:43.971647 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0804 14:51:43.971666 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:51:43.971736 (+ 70us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:51:43.971771 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:51:43.971774 (+ 3us) write_op.cc:312] APPLY: finished
0804 14:51:43.971838 (+ 64us) log.cc:842] Serialized 22 byte log entry
0804 14:51:43.971950 (+ 112us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:43.971964 (+ 14us) write_op.cc:454] Released schema lock
0804 14:51:43.971969 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":230,"num_ops":1,"prepare.queue_time_us":211978,"prepare.run_cpu_time_us":782,"prepare.run_wall_time_us":552868,"raft.queue_time_us":37,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":252,"replication_time_us":692791,"thread_start_us":68,"threads_started":1}]]}
W20260804 14:51:43.973510 27335 scanner-internal.cc:458] Time spent opening tablet: real 0.797s user 0.001s sys 0.000s
W20260804 14:51:43.973958 27336 scanner-internal.cc:458] Time spent opening tablet: real 0.798s user 0.000s sys 0.001s
I20260804 14:51:44.177776 27471 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:44.305876 27471 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:44.306067 27471 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:44.306627 27471 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:44.307080 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:44.307075 26844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:44.307549 27528 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50938 (ReqId={client: 2f11fb73142a45a5ab6d1317d6b28d6e, seq_no=27, attempt_no=0}) took 1282 ms. Trace:
I20260804 14:51:44.307654 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:44.307636 27528 rpcz_store.cc:276] 0804 14:51:43.025557 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:43.025633 (+ 76us) service_pool.cc:227] Handling call
0804 14:51:44.307538 (+1281905us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:43.278824 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:43.278942 (+ 118us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:43.278947 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:51:43.278949 (+ 2us) tablet.cc:660] Decoding operations
0804 14:51:43.279033 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:43.279042 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:43.279045 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:43.279093 (+ 48us) tablet.cc:699] Row locks acquired
0804 14:51:43.279096 (+ 3us) write_op.cc:260] PREPARE: finished
0804 14:51:43.279144 (+ 48us) write_op.cc:270] Start()
0804 14:51:44.243320 (+964176us) write_op.cc:276] Timestamp: P: 1785855103279091 usec, L: 0
0804 14:51:44.243324 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:51:44.243585 (+ 261us) log.cc:842] Serialized 354 byte log entry
0804 14:51:44.306907 (+ 63322us) op_driver.cc:444] REPLICATION: finished
0804 14:51:44.306990 (+ 83us) write_op.cc:301] APPLY: starting
0804 14:51:44.307007 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0804 14:51:44.307047 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0804 14:51:44.307129 (+ 82us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:51:44.307220 (+ 91us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:51:44.307251 (+ 31us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:51:44.307256 (+ 5us) write_op.cc:312] APPLY: finished
0804 14:51:44.307345 (+ 89us) log.cc:842] Serialized 22 byte log entry
0804 14:51:44.307438 (+ 93us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:44.307453 (+ 15us) write_op.cc:454] Released schema lock
0804 14:51:44.307458 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":34,"num_ops":1,"prepare.queue_time_us":252785,"prepare.run_cpu_time_us":757,"prepare.run_wall_time_us":964824,"replication_time_us":63557}]]}
I20260804 14:51:44.308640 27471 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:44.817163 27471 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:44.817327 27471 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:44.817866 27471 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:44.818356 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:44.818375 26844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:44.819422 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:44.824024 27482 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:44.824187 27482 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:44.824694 27482 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:44.825033 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:44.825243 27199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:44.825655 26755 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:45.286252 27471 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260804 14:51:45.289291 27336 scanner-internal.cc:458] Time spent opening tablet: real 0.830s user 0.001s sys 0.000s
W20260804 14:51:45.289324 27335 scanner-internal.cc:458] Time spent opening tablet: real 0.830s user 0.001s sys 0.000s
I20260804 14:51:45.428673 27482 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:45.670099 27471 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:45.670261 27471 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:45.670871 27471 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:45.671332 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:45.671355 26844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:45.671932 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:45.679939 27471 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:46.287367 27482 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:46.287541 27482 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:46.288019 27482 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:46.288354 27197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:46.289074 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:46.289476 26757 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:46.289646 27482 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:46.335304 27482 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:46.335474 27482 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:46.335952 27482 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:46.336284 27197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:46.336519 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:46.336901 26757 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:46.337122 27482 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:46.826479 27482 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:46.826676 27482 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:46.827229 27482 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:46.827685 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:46.827706 27199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:46.828205 26755 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:46.828392 27482 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:46.828823 27551 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50938 (ReqId={client: 2f11fb73142a45a5ab6d1317d6b28d6e, seq_no=32, attempt_no=0}) took 1155 ms. Trace:
I20260804 14:51:46.828943 27551 rpcz_store.cc:276] 0804 14:51:45.672911 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:45.673010 (+ 99us) service_pool.cc:227] Handling call
0804 14:51:46.828805 (+1155795us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:45.673350 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:45.673424 (+ 74us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:45.673427 (+ 3us) write_op.cc:435] Acquired schema lock
0804 14:51:45.673429 (+ 2us) tablet.cc:660] Decoding operations
0804 14:51:45.673488 (+ 59us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:45.673495 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:45.673499 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:45.673528 (+ 29us) tablet.cc:699] Row locks acquired
0804 14:51:45.673528 (+ 0us) write_op.cc:260] PREPARE: finished
0804 14:51:45.673555 (+ 27us) write_op.cc:270] Start()
0804 14:51:46.600725 (+927170us) write_op.cc:276] Timestamp: P: 1785855105673516 usec, L: 0
0804 14:51:46.600729 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:51:46.601095 (+ 366us) log.cc:842] Serialized 354 byte log entry
0804 14:51:46.828050 (+226955us) op_driver.cc:444] REPLICATION: finished
0804 14:51:46.828309 (+ 259us) write_op.cc:301] APPLY: starting
0804 14:51:46.828329 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0804 14:51:46.828365 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0804 14:51:46.828381 (+ 16us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:51:46.828451 (+ 70us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:51:46.828485 (+ 34us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:51:46.828490 (+ 5us) write_op.cc:312] APPLY: finished
0804 14:51:46.828597 (+ 107us) log.cc:842] Serialized 22 byte log entry
0804 14:51:46.828704 (+ 107us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:46.828718 (+ 14us) write_op.cc:454] Released schema lock
0804 14:51:46.828723 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":204,"num_ops":1,"prepare.queue_time_us":43,"prepare.run_cpu_time_us":739,"prepare.run_wall_time_us":927809,"replication_time_us":227292,"spinlock_wait_cycles":2944,"thread_start_us":76,"threads_started":1}]]}
W20260804 14:51:46.830312 27336 scanner-internal.cc:458] Time spent opening tablet: real 0.997s user 0.001s sys 0.000s
W20260804 14:51:46.830312 27335 scanner-internal.cc:458] Time spent opening tablet: real 0.997s user 0.000s sys 0.000s
I20260804 14:51:47.078959 27547 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:47.079128 27547 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:47.079840 27547 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:47.080355 26844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:47.080389 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:47.081262 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:47.805927 27482 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:47.806109 27482 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:47.806718 27482 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:47.807142 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:47.807730 27199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:47.808264 26755 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:47.808581 27482 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:47.809516 27562 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50938 (ReqId={client: 2f11fb73142a45a5ab6d1317d6b28d6e, seq_no=33, attempt_no=0}) took 2126 ms. Trace:
I20260804 14:51:47.809656 27562 rpcz_store.cc:276] 0804 14:51:45.682855 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:45.682904 (+ 49us) service_pool.cc:227] Handling call
0804 14:51:47.809498 (+2126594us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:46.601200 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:46.601310 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:46.601317 (+ 7us) write_op.cc:435] Acquired schema lock
0804 14:51:46.601319 (+ 2us) tablet.cc:660] Decoding operations
0804 14:51:46.601427 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:46.601436 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:46.601438 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:46.601478 (+ 40us) tablet.cc:699] Row locks acquired
0804 14:51:46.601479 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:51:46.601511 (+ 32us) write_op.cc:270] Start()
0804 14:51:46.759747 (+158236us) write_op.cc:276] Timestamp: P: 1785855106601492 usec, L: 0
0804 14:51:46.759751 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:51:46.759973 (+ 222us) log.cc:842] Serialized 354 byte log entry
0804 14:51:47.808638 (+1048665us) op_driver.cc:444] REPLICATION: finished
0804 14:51:47.808916 (+ 278us) write_op.cc:301] APPLY: starting
0804 14:51:47.808940 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
0804 14:51:47.808979 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0804 14:51:47.808995 (+ 16us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:51:47.809098 (+ 103us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:51:47.809135 (+ 37us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:51:47.809141 (+ 6us) write_op.cc:312] APPLY: finished
0804 14:51:47.809218 (+ 77us) log.cc:842] Serialized 22 byte log entry
0804 14:51:47.809361 (+ 143us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:47.809391 (+ 30us) write_op.cc:454] Released schema lock
0804 14:51:47.809399 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":181,"num_ops":1,"prepare.queue_time_us":917992,"prepare.run_cpu_time_us":759,"prepare.run_wall_time_us":158876,"replication_time_us":1048860,"thread_start_us":105,"threads_started":1}]]}
I20260804 14:51:48.206568 27547 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:48.207135 27197 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.24.6.130:33363 (request call id 3984) took 1126 ms. Trace:
I20260804 14:51:48.207274 27197 rpcz_store.cc:276] 0804 14:51:47.080478 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:47.080539 (+ 61us) service_pool.cc:227] Handling call
0804 14:51:47.080689 (+ 150us) raft_consensus.cc:1457] Updating replica for 2 ops
0804 14:51:47.080775 (+ 86us) raft_consensus.cc:1520] Early marking committed up to index 34
0804 14:51:47.080923 (+ 148us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0804 14:51:47.081064 (+ 141us) write_op.cc:270] Start()
0804 14:51:47.317323 (+236259us) write_op.cc:276] Timestamp: P: 1785855106601492 usec, L: 0
0804 14:51:47.317688 (+ 365us) write_op.cc:270] Start()
0804 14:51:48.205863 (+888175us) write_op.cc:276] Timestamp: P: 1785855106831323 usec, L: 0
0804 14:51:48.206298 (+ 435us) log.cc:842] Serialized 708 byte log entry
0804 14:51:48.206454 (+ 156us) raft_consensus.cc:1631] Marking committed up to 34
0804 14:51:48.206460 (+ 6us) raft_consensus.cc:1683] Filling consensus response to leader.
0804 14:51:48.206495 (+ 35us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0804 14:51:48.207065 (+ 570us) raft_consensus.cc:1671] finished
0804 14:51:48.207068 (+ 3us) raft_consensus.cc:1677] UpdateReplicas() finished
0804 14:51:48.207119 (+ 51us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:47.317560 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:47.317687 (+ 127us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:47.317691 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:51:47.317692 (+ 1us) tablet.cc:660] Decoding operations
0804 14:51:47.317789 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:47.317800 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:47.317803 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:47.317843 (+ 40us) tablet.cc:699] Row locks acquired
0804 14:51:47.317844 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0804 14:51:48.206263 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:48.206363 (+ 100us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:48.206367 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:51:48.206369 (+ 2us) tablet.cc:660] Decoding operations
0804 14:51:48.206485 (+ 116us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:48.206500 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:48.206503 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:48.206555 (+ 52us) tablet.cc:699] Row locks acquired
0804 14:51:48.206556 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":104,"threads_started":1,"wal-append.queue_time_us":164,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":77,"prepare.run_cpu_time_us":365,"prepare.run_wall_time_us":365}],["op",{"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":247,"prepare.run_cpu_time_us":401,"prepare.run_wall_time_us":402,"thread_start_us":86,"threads_started":1}]]}
I20260804 14:51:48.209358 27562 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50938 (ReqId={client: 2f11fb73142a45a5ab6d1317d6b28d6e, seq_no=34, attempt_no=0}) took 1378 ms. Trace:
I20260804 14:51:48.209447 27562 rpcz_store.cc:276] 0804 14:51:46.830638 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:46.830687 (+ 49us) service_pool.cc:227] Handling call
0804 14:51:48.209346 (+1378659us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:46.831086 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:46.831164 (+ 78us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:46.831167 (+ 3us) write_op.cc:435] Acquired schema lock
0804 14:51:46.831168 (+ 1us) tablet.cc:660] Decoding operations
0804 14:51:46.831281 (+ 113us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:46.831290 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:46.831293 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:46.831332 (+ 39us) tablet.cc:699] Row locks acquired
0804 14:51:46.831333 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:51:46.831363 (+ 30us) write_op.cc:270] Start()
0804 14:51:47.058567 (+227204us) write_op.cc:276] Timestamp: P: 1785855106831323 usec, L: 0
0804 14:51:47.058573 (+ 6us) op_driver.cc:331] REPLICATION: starting
0804 14:51:47.058982 (+ 409us) log.cc:842] Serialized 354 byte log entry
0804 14:51:48.208844 (+1149862us) op_driver.cc:444] REPLICATION: finished
0804 14:51:48.208948 (+ 104us) write_op.cc:301] APPLY: starting
0804 14:51:48.208968 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0804 14:51:48.209001 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0804 14:51:48.209022 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:51:48.209075 (+ 53us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:51:48.209100 (+ 25us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:51:48.209104 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:51:48.209163 (+ 59us) log.cc:842] Serialized 22 byte log entry
0804 14:51:48.209250 (+ 87us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:48.209265 (+ 15us) write_op.cc:454] Released schema lock
0804 14:51:48.209272 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":56,"num_ops":1,"prepare.queue_time_us":50,"prepare.run_cpu_time_us":929,"prepare.run_wall_time_us":228016,"replication_time_us":1150245}]]}
I20260804 14:51:48.210497 26967 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50938 (request call id 79) took 1228 ms. Trace:
I20260804 14:51:48.210597 26967 rpcz_store.cc:276] 0804 14:51:46.981816 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:46.981917 (+ 101us) service_pool.cc:227] Handling call
0804 14:51:46.982181 (+ 264us) tablet_service.cc:2780] Created scanner 05c347724ef94167b62213e73e478c9e for tablet df07e91451bc49ba978359f97a315ebe, query id is 2f1e0593c7a448deb280489c888a4c4f
0804 14:51:46.982382 (+ 201us) tablet_service.cc:2920] Creating iterator
0804 14:51:46.982411 (+ 29us) tablet_service.cc:3297] Waiting safe time to advance
0804 14:51:47.058779 (+ 76368us) tablet_service.cc:3304] Waiting for operations to commit
0804 14:51:48.209324 (+1150545us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1226897 microseconds
0804 14:51:48.209440 (+ 116us) tablet_service.cc:2945] Iterator created
0804 14:51:48.209860 (+ 420us) tablet_service.cc:2967] Iterator init: OK
0804 14:51:48.209894 (+ 34us) tablet_service.cc:3010] has_more: true
0804 14:51:48.209930 (+ 36us) tablet_service.cc:3027] Continuing scan request
0804 14:51:48.210006 (+ 76us) tablet_service.cc:3091] Found scanner 05c347724ef94167b62213e73e478c9e for tablet df07e91451bc49ba978359f97a315ebe, query id is 2f1e0593c7a448deb280489c888a4c4f
0804 14:51:48.210486 (+ 480us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":896}
I20260804 14:51:48.210719 26964 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50938 (request call id 80) took 1228 ms. Trace:
I20260804 14:51:48.210813 26964 rpcz_store.cc:276] 0804 14:51:46.981951 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:46.982057 (+ 106us) service_pool.cc:227] Handling call
0804 14:51:46.982190 (+ 133us) tablet_service.cc:2780] Created scanner ed5e08a210244e6a911e89ff2e8357fe for tablet df07e91451bc49ba978359f97a315ebe, query id is bb7cae9a837549d49c6ddf02d8d71c1d
0804 14:51:46.982434 (+ 244us) tablet_service.cc:2920] Creating iterator
0804 14:51:46.982460 (+ 26us) tablet_service.cc:3297] Waiting safe time to advance
0804 14:51:47.058792 (+ 76332us) tablet_service.cc:3304] Waiting for operations to commit
0804 14:51:48.209923 (+1151131us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1227455 microseconds
0804 14:51:48.209992 (+ 69us) tablet_service.cc:2945] Iterator created
0804 14:51:48.210266 (+ 274us) tablet_service.cc:2967] Iterator init: OK
0804 14:51:48.210290 (+ 24us) tablet_service.cc:3010] has_more: true
0804 14:51:48.210317 (+ 27us) tablet_service.cc:3027] Continuing scan request
0804 14:51:48.210356 (+ 39us) tablet_service.cc:3091] Found scanner ed5e08a210244e6a911e89ff2e8357fe for tablet df07e91451bc49ba978359f97a315ebe, query id is bb7cae9a837549d49c6ddf02d8d71c1d
0804 14:51:48.210709 (+ 353us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":384}
W20260804 14:51:48.211633 27335 scanner-internal.cc:458] Time spent opening tablet: real 1.231s user 0.001s sys 0.000s
W20260804 14:51:48.211635 27336 scanner-internal.cc:458] Time spent opening tablet: real 1.231s user 0.001s sys 0.000s
I20260804 14:51:48.410399 27482 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:48.410581 27482 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:48.411247 27482 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:48.411579 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:48.412212 27197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:48.412636 26755 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:48.412904 27482 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:48.740206 27482 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:48.740370 27482 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:48.740895 27482 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:48.741254 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:48.741607 27197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:48.742694 26755 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:48.742938 27482 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:48.909653 27547 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:48.909802 27547 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:48.910295 27547 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:48.910756 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:48.911320 26845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:48.911746 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:48.911960 27547 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:49.978068 27581 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:49.978235 27581 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:49.978883 27581 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:49.979288 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:49.979589 27197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:49.980634 26755 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:49.980671 26967 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50938 (request call id 87) took 1085 ms. Trace:
I20260804 14:51:49.980796 26964 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50938 (request call id 86) took 1086 ms. Trace:
I20260804 14:51:49.980510 27585 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50938 (ReqId={client: 2f11fb73142a45a5ab6d1317d6b28d6e, seq_no=36, attempt_no=0}) took 1768 ms. Trace:
I20260804 14:51:49.980844 26964 rpcz_store.cc:276] 0804 14:51:48.894599 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:48.894672 (+ 73us) service_pool.cc:227] Handling call
0804 14:51:48.894850 (+ 178us) tablet_service.cc:2780] Created scanner b407e9457dca47a6bd75f036a8df2fdb for tablet df07e91451bc49ba978359f97a315ebe, query id is 2dacb5631bd84ebb8254ba2484eb60e8
0804 14:51:48.895128 (+ 278us) tablet_service.cc:2920] Creating iterator
0804 14:51:48.895165 (+ 37us) tablet_service.cc:3297] Waiting safe time to advance
0804 14:51:49.615926 (+720761us) tablet_service.cc:3304] Waiting for operations to commit
0804 14:51:49.979720 (+363794us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1084544 microseconds
0804 14:51:49.979809 (+ 89us) tablet_service.cc:2945] Iterator created
0804 14:51:49.980146 (+ 337us) tablet_service.cc:2967] Iterator init: OK
0804 14:51:49.980176 (+ 30us) tablet_service.cc:3010] has_more: true
0804 14:51:49.980204 (+ 28us) tablet_service.cc:3027] Continuing scan request
0804 14:51:49.980257 (+ 53us) tablet_service.cc:3091] Found scanner b407e9457dca47a6bd75f036a8df2fdb for tablet df07e91451bc49ba978359f97a315ebe, query id is 2dacb5631bd84ebb8254ba2484eb60e8
0804 14:51:49.980790 (+ 533us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":18688}
I20260804 14:51:49.981033 27581 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:49.980741 26967 rpcz_store.cc:276] 0804 14:51:48.894697 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:48.894742 (+ 45us) service_pool.cc:227] Handling call
0804 14:51:48.894878 (+ 136us) tablet_service.cc:2780] Created scanner c3280eb1de6e4086ab2cb924329f2f6b for tablet df07e91451bc49ba978359f97a315ebe, query id is 12fdcd635e1a442dab24701d98a9874f
0804 14:51:48.895098 (+ 220us) tablet_service.cc:2920] Creating iterator
0804 14:51:48.895132 (+ 34us) tablet_service.cc:3297] Waiting safe time to advance
0804 14:51:49.615926 (+720794us) tablet_service.cc:3304] Waiting for operations to commit
0804 14:51:49.979720 (+363794us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1084580 microseconds
0804 14:51:49.979809 (+ 89us) tablet_service.cc:2945] Iterator created
0804 14:51:49.980146 (+ 337us) tablet_service.cc:2967] Iterator init: OK
0804 14:51:49.980172 (+ 26us) tablet_service.cc:3010] has_more: true
0804 14:51:49.980204 (+ 32us) tablet_service.cc:3027] Continuing scan request
0804 14:51:49.980257 (+ 53us) tablet_service.cc:3091] Found scanner c3280eb1de6e4086ab2cb924329f2f6b for tablet df07e91451bc49ba978359f97a315ebe, query id is 12fdcd635e1a442dab24701d98a9874f
0804 14:51:49.980659 (+ 402us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":37760}
I20260804 14:51:49.980926 27585 rpcz_store.cc:276] 0804 14:51:48.212102 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:48.212178 (+ 76us) service_pool.cc:227] Handling call
0804 14:51:49.980495 (+1768317us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:48.682264 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:48.682377 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:48.682391 (+ 14us) write_op.cc:435] Acquired schema lock
0804 14:51:48.682393 (+ 2us) tablet.cc:660] Decoding operations
0804 14:51:48.682479 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:48.682489 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:48.682492 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:48.682541 (+ 49us) tablet.cc:699] Row locks acquired
0804 14:51:48.682543 (+ 2us) write_op.cc:260] PREPARE: finished
0804 14:51:48.682594 (+ 51us) write_op.cc:270] Start()
0804 14:51:49.615717 (+933123us) write_op.cc:276] Timestamp: P: 1785855108682565 usec, L: 0
0804 14:51:49.615720 (+ 3us) op_driver.cc:331] REPLICATION: starting
0804 14:51:49.616034 (+ 314us) log.cc:842] Serialized 354 byte log entry
0804 14:51:49.979079 (+363045us) op_driver.cc:444] REPLICATION: finished
0804 14:51:49.979336 (+ 257us) write_op.cc:301] APPLY: starting
0804 14:51:49.979354 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0804 14:51:49.979388 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0804 14:51:49.979399 (+ 11us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:51:49.979460 (+ 61us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:51:49.979519 (+ 59us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:51:49.979526 (+ 7us) write_op.cc:312] APPLY: finished
0804 14:51:49.979605 (+ 79us) log.cc:842] Serialized 22 byte log entry
0804 14:51:49.980382 (+ 777us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:49.980400 (+ 18us) write_op.cc:454] Released schema lock
0804 14:51:49.980408 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":206,"mutex_wait_us":15,"num_ops":1,"prepare.queue_time_us":469722,"prepare.run_cpu_time_us":800,"prepare.run_wall_time_us":933854,"raft.queue_time_us":187,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":263,"replication_time_us":363335,"spinlock_wait_cycles":36096,"thread_start_us":91,"threads_started":1}]]}
W20260804 14:51:49.981449 27335 scanner-internal.cc:458] Time spent opening tablet: real 1.088s user 0.001s sys 0.000s
W20260804 14:51:49.981487 27336 scanner-internal.cc:458] Time spent opening tablet: real 1.088s user 0.001s sys 0.000s
I20260804 14:51:50.345826 27580 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:50.345979 27580 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:50.346540 27580 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:50.346907 26844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:50.347064 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:50.347496 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:50.347752 27580 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:50.735850 27580 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:50.736057 27580 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:50.736706 27580 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:50.737177 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:50.737248 26845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:50.737730 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:50.737998 27580 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260804 14:51:50.739339 27336 scanner-internal.cc:458] Time spent opening tablet: real 0.607s user 0.001s sys 0.000s
I20260804 14:51:50.739441 27594 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50938 (ReqId={client: 2f11fb73142a45a5ab6d1317d6b28d6e, seq_no=37, attempt_no=0}) took 1995 ms. Trace:
I20260804 14:51:50.739522 27594 rpcz_store.cc:276] 0804 14:51:48.744139 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:48.744202 (+ 63us) service_pool.cc:227] Handling call
0804 14:51:50.739421 (+1995219us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:49.616149 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:49.616256 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:49.616261 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:51:49.616262 (+ 1us) tablet.cc:660] Decoding operations
0804 14:51:49.616343 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:49.616351 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:49.616354 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:49.616394 (+ 40us) tablet.cc:699] Row locks acquired
0804 14:51:49.616395 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:51:49.616453 (+ 58us) write_op.cc:270] Start()
0804 14:51:50.477616 (+861163us) write_op.cc:276] Timestamp: P: 1785855109616419 usec, L: 0
0804 14:51:50.477622 (+ 6us) op_driver.cc:331] REPLICATION: starting
0804 14:51:50.477859 (+ 237us) log.cc:842] Serialized 354 byte log entry
0804 14:51:50.736937 (+259078us) op_driver.cc:444] REPLICATION: finished
0804 14:51:50.737257 (+ 320us) write_op.cc:301] APPLY: starting
0804 14:51:50.737278 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0804 14:51:50.737314 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0804 14:51:50.737328 (+ 14us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:51:50.737394 (+ 66us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:51:50.737434 (+ 40us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:51:50.737439 (+ 5us) write_op.cc:312] APPLY: finished
0804 14:51:50.737672 (+ 233us) log.cc:842] Serialized 22 byte log entry
0804 14:51:50.738912 (+ 1240us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:50.738929 (+ 17us) write_op.cc:454] Released schema lock
0804 14:51:50.738936 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":258,"num_ops":1,"prepare.queue_time_us":871644,"prepare.run_cpu_time_us":699,"prepare.run_wall_time_us":861773,"replication_time_us":259284,"thread_start_us":90,"threads_started":1}]]}
W20260804 14:51:50.739339 27335 scanner-internal.cc:458] Time spent opening tablet: real 0.607s user 0.001s sys 0.000s
I20260804 14:51:50.764247 27593 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:50.764407 27593 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:50.765273 27593 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:50.765295 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:50.765372 27197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:50.765769 26757 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:50.765988 27593 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:50.895112 27580 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:50.895309 27580 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:50.895900 27580 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:50.896343 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:50.896343 26845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:50.896847 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:50.897440 27580 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:51.634687 27593 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:51.634861 27593 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:51.635510 27593 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:51.635978 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:51.636406 27197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:51.636955 26755 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:51.723876 27580 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:51.724023 27580 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:51.724567 27580 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:51.725019 26845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:51.725026 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:51.725561 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:51.726709 27580 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260804 14:51:51.727224 27335 scanner-internal.cc:458] Time spent opening tablet: real 0.837s user 0.001s sys 0.000s
W20260804 14:51:51.727315 27336 scanner-internal.cc:458] Time spent opening tablet: real 0.837s user 0.000s sys 0.001s
I20260804 14:51:51.797451 27593 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:52.106215 27580 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:52.106379 27580 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:52.106978 27580 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:52.107481 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:52.107470 26845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:52.107777 27603 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50938 (ReqId={client: 2f11fb73142a45a5ab6d1317d6b28d6e, seq_no=40, attempt_no=0}) took 1209 ms. Trace:
I20260804 14:51:52.107861 27603 rpcz_store.cc:276] 0804 14:51:50.898312 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:50.898385 (+ 73us) service_pool.cc:227] Handling call
0804 14:51:52.107765 (+1209380us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:50.970432 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:50.970542 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:50.970548 (+ 6us) write_op.cc:435] Acquired schema lock
0804 14:51:50.970549 (+ 1us) tablet.cc:660] Decoding operations
0804 14:51:50.970648 (+ 99us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:50.970660 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:50.970663 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:50.970709 (+ 46us) tablet.cc:699] Row locks acquired
0804 14:51:50.970710 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:51:50.970753 (+ 43us) write_op.cc:270] Start()
0804 14:51:51.867432 (+896679us) write_op.cc:276] Timestamp: P: 1785855110970702 usec, L: 0
0804 14:51:51.867435 (+ 3us) op_driver.cc:331] REPLICATION: starting
0804 14:51:51.867645 (+ 210us) log.cc:842] Serialized 354 byte log entry
0804 14:51:52.107289 (+239644us) op_driver.cc:444] REPLICATION: finished
0804 14:51:52.107377 (+ 88us) write_op.cc:301] APPLY: starting
0804 14:51:52.107398 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0804 14:51:52.107434 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0804 14:51:52.107449 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:51:52.107511 (+ 62us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:51:52.107543 (+ 32us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:51:52.107547 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:51:52.107614 (+ 67us) log.cc:842] Serialized 22 byte log entry
0804 14:51:52.107673 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:52.107687 (+ 14us) write_op.cc:454] Released schema lock
0804 14:51:52.107694 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":33,"num_ops":1,"prepare.queue_time_us":71738,"prepare.run_cpu_time_us":716,"prepare.run_wall_time_us":897308,"raft.queue_time_us":42,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":204,"replication_time_us":239824}]]}
I20260804 14:51:52.108287 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:52.108460 27580 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:52.514833 27593 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:52.514990 27593 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:52.515599 27593 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:52.516007 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:52.516791 27199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:52.517277 26755 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:52.517467 27593 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:53.411782 27593 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:53.411986 27593 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:53.412547 27593 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:53.412936 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:53.413259 27197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:53.413918 26757 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:53.414412 27593 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:53.414458 27611 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50938 (ReqId={client: 2f11fb73142a45a5ab6d1317d6b28d6e, seq_no=41, attempt_no=0}) took 1687 ms. Trace:
I20260804 14:51:53.414640 27611 rpcz_store.cc:276] 0804 14:51:51.726894 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:51.726953 (+ 59us) service_pool.cc:227] Handling call
0804 14:51:53.414440 (+1687487us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:51.867777 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:51.867929 (+ 152us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:51.867937 (+ 8us) write_op.cc:435] Acquired schema lock
0804 14:51:51.867939 (+ 2us) tablet.cc:660] Decoding operations
0804 14:51:51.868021 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:51.868030 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:51.868032 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:51.868071 (+ 39us) tablet.cc:699] Row locks acquired
0804 14:51:51.868072 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:51:51.868130 (+ 58us) write_op.cc:270] Start()
0804 14:51:52.552267 (+684137us) write_op.cc:276] Timestamp: P: 1785855111868030 usec, L: 0
0804 14:51:52.552270 (+ 3us) op_driver.cc:331] REPLICATION: starting
0804 14:51:52.552544 (+ 274us) log.cc:842] Serialized 354 byte log entry
0804 14:51:53.413609 (+861065us) op_driver.cc:444] REPLICATION: finished
0804 14:51:53.413908 (+ 299us) write_op.cc:301] APPLY: starting
0804 14:51:53.413935 (+ 27us) tablet.cc:1365] starting BulkCheckPresence
0804 14:51:53.414006 (+ 71us) tablet.cc:1368] finished BulkCheckPresence
0804 14:51:53.414022 (+ 16us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:51:53.414118 (+ 96us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:51:53.414151 (+ 33us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:51:53.414160 (+ 9us) write_op.cc:312] APPLY: finished
0804 14:51:53.414234 (+ 74us) log.cc:842] Serialized 22 byte log entry
0804 14:51:53.414342 (+ 108us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:53.414356 (+ 14us) write_op.cc:454] Released schema lock
0804 14:51:53.414362 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":238,"num_ops":1,"prepare.queue_time_us":140424,"prepare.run_cpu_time_us":767,"prepare.run_wall_time_us":684832,"replication_time_us":861314,"thread_start_us":59,"threads_started":1}]]}
I20260804 14:51:53.415501 26967 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50938 (request call id 97) took 1536 ms. Trace:
I20260804 14:51:53.415607 26964 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50938 (request call id 96) took 1537 ms. Trace:
I20260804 14:51:53.415570 26967 rpcz_store.cc:276] 0804 14:51:51.878619 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:51.878677 (+ 58us) service_pool.cc:227] Handling call
0804 14:51:51.878805 (+ 128us) tablet_service.cc:2780] Created scanner 0fb5dd965aa54bd583415afde9de0959 for tablet df07e91451bc49ba978359f97a315ebe, query id is 56905022908f44f8ba68dbdac277cdf6
0804 14:51:51.878965 (+ 160us) tablet_service.cc:2920] Creating iterator
0804 14:51:51.879010 (+ 45us) tablet_service.cc:3297] Waiting safe time to advance
0804 14:51:52.552411 (+673401us) tablet_service.cc:3304] Waiting for operations to commit
0804 14:51:53.414469 (+862058us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1535449 microseconds
0804 14:51:53.414568 (+ 99us) tablet_service.cc:2945] Iterator created
0804 14:51:53.414931 (+ 363us) tablet_service.cc:2967] Iterator init: OK
0804 14:51:53.414956 (+ 25us) tablet_service.cc:3010] has_more: true
0804 14:51:53.414999 (+ 43us) tablet_service.cc:3027] Continuing scan request
0804 14:51:53.415070 (+ 71us) tablet_service.cc:3091] Found scanner 0fb5dd965aa54bd583415afde9de0959 for tablet df07e91451bc49ba978359f97a315ebe, query id is 56905022908f44f8ba68dbdac277cdf6
0804 14:51:53.415490 (+ 420us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":27392}
I20260804 14:51:53.415704 26964 rpcz_store.cc:276] 0804 14:51:51.878541 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:51.878607 (+ 66us) service_pool.cc:227] Handling call
0804 14:51:51.878781 (+ 174us) tablet_service.cc:2780] Created scanner 93bf8dda64d94d27a9fd85ae64143654 for tablet df07e91451bc49ba978359f97a315ebe, query id is fa448186ec794600a35270cc4fbd88e8
0804 14:51:51.878967 (+ 186us) tablet_service.cc:2920] Creating iterator
0804 14:51:51.879012 (+ 45us) tablet_service.cc:3297] Waiting safe time to advance
0804 14:51:52.552384 (+673372us) tablet_service.cc:3304] Waiting for operations to commit
0804 14:51:53.414633 (+862249us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1535606 microseconds
0804 14:51:53.414720 (+ 87us) tablet_service.cc:2945] Iterator created
0804 14:51:53.415059 (+ 339us) tablet_service.cc:2967] Iterator init: OK
0804 14:51:53.415085 (+ 26us) tablet_service.cc:3010] has_more: true
0804 14:51:53.415117 (+ 32us) tablet_service.cc:3027] Continuing scan request
0804 14:51:53.415167 (+ 50us) tablet_service.cc:3091] Found scanner 93bf8dda64d94d27a9fd85ae64143654 for tablet df07e91451bc49ba978359f97a315ebe, query id is fa448186ec794600a35270cc4fbd88e8
0804 14:51:53.415595 (+ 428us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":32384}
W20260804 14:51:53.416354 27335 scanner-internal.cc:458] Time spent opening tablet: real 1.539s user 0.001s sys 0.000s
W20260804 14:51:53.416379 27336 scanner-internal.cc:458] Time spent opening tablet: real 1.539s user 0.000s sys 0.001s
I20260804 14:51:53.550251 27580 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:53.550395 27580 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:53.550971 27580 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:53.551617 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:53.551824 26845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:53.552366 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:53.807932 27593 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:53.808096 27593 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:53.808712 27593 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:53.809067 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 44 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:53.809245 27197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 44 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:53.809513 27611 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50938 (ReqId={client: 2f11fb73142a45a5ab6d1317d6b28d6e, seq_no=42, attempt_no=0}) took 1700 ms. Trace:
I20260804 14:51:53.809588 27611 rpcz_store.cc:276] 0804 14:51:52.109448 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:52.109519 (+ 71us) service_pool.cc:227] Handling call
0804 14:51:53.809502 (+1699983us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:52.552656 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:52.552765 (+ 109us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:52.552770 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:51:52.552771 (+ 1us) tablet.cc:660] Decoding operations
0804 14:51:52.552858 (+ 87us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:52.552868 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:52.552870 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:52.552915 (+ 45us) tablet.cc:699] Row locks acquired
0804 14:51:52.552917 (+ 2us) write_op.cc:260] PREPARE: finished
0804 14:51:52.552961 (+ 44us) write_op.cc:270] Start()
0804 14:51:53.512116 (+959155us) write_op.cc:276] Timestamp: P: 1785855112552899 usec, L: 0
0804 14:51:53.512119 (+ 3us) op_driver.cc:331] REPLICATION: starting
0804 14:51:53.512313 (+ 194us) log.cc:842] Serialized 354 byte log entry
0804 14:51:53.808992 (+296679us) op_driver.cc:444] REPLICATION: finished
0804 14:51:53.809101 (+ 109us) write_op.cc:301] APPLY: starting
0804 14:51:53.809120 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0804 14:51:53.809152 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0804 14:51:53.809166 (+ 14us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:51:53.809237 (+ 71us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:51:53.809267 (+ 30us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:51:53.809271 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:51:53.809343 (+ 72us) log.cc:842] Serialized 22 byte log entry
0804 14:51:53.809401 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:53.809416 (+ 15us) write_op.cc:454] Released schema lock
0804 14:51:53.809421 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":61,"num_ops":1,"prepare.queue_time_us":442786,"prepare.run_cpu_time_us":684,"prepare.run_wall_time_us":959759,"raft.queue_time_us":52,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":128,"replication_time_us":296848,"spinlock_wait_cycles":46976}]]}
I20260804 14:51:53.809857 26755 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:53.810427 27593 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:54.337119 27593 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:54.337308 27593 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:54.337839 27593 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:54.338336 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:54.338342 27197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:54.338884 26755 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
W20260804 14:51:54.341154 27336 scanner-internal.cc:458] Time spent opening tablet: real 0.774s user 0.000s sys 0.000s
W20260804 14:51:54.341198 27335 scanner-internal.cc:458] Time spent opening tablet: real 0.774s user 0.001s sys 0.000s
I20260804 14:51:54.509502 27580 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:55.103283 27593 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:55.105255 27631 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50938 (ReqId={client: 2f11fb73142a45a5ab6d1317d6b28d6e, seq_no=44, attempt_no=0}) took 1293 ms. Trace:
I20260804 14:51:55.105396 27631 rpcz_store.cc:276] 0804 14:51:53.811462 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:53.811546 (+ 84us) service_pool.cc:227] Handling call
0804 14:51:55.105235 (+1293689us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:54.176531 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:54.176655 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:54.176661 (+ 6us) write_op.cc:435] Acquired schema lock
0804 14:51:54.176663 (+ 2us) tablet.cc:660] Decoding operations
0804 14:51:54.176751 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:54.176787 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:54.176793 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:54.176845 (+ 52us) tablet.cc:699] Row locks acquired
0804 14:51:54.176847 (+ 2us) write_op.cc:260] PREPARE: finished
0804 14:51:54.176887 (+ 40us) write_op.cc:270] Start()
0804 14:51:54.229088 (+ 52201us) write_op.cc:276] Timestamp: P: 1785855114176831 usec, L: 0
0804 14:51:54.229092 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:51:54.229370 (+ 278us) log.cc:842] Serialized 354 byte log entry
0804 14:51:55.104536 (+875166us) op_driver.cc:444] REPLICATION: finished
0804 14:51:55.104812 (+ 276us) write_op.cc:301] APPLY: starting
0804 14:51:55.104832 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0804 14:51:55.104866 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0804 14:51:55.104879 (+ 13us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:51:55.104945 (+ 66us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:51:55.104981 (+ 36us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:51:55.104984 (+ 3us) write_op.cc:312] APPLY: finished
0804 14:51:55.105049 (+ 65us) log.cc:842] Serialized 22 byte log entry
0804 14:51:55.105137 (+ 88us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:55.105151 (+ 14us) write_op.cc:454] Released schema lock
0804 14:51:55.105156 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":210,"num_ops":1,"prepare.queue_time_us":364478,"prepare.run_cpu_time_us":859,"prepare.run_wall_time_us":52929,"replication_time_us":875392,"thread_start_us":101,"threads_started":1}]]}
I20260804 14:51:55.369148 27593 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:55.369316 27593 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:55.369870 27593 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:55.370335 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:55.370467 27197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:55.370937 27631 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50938 (ReqId={client: 2f11fb73142a45a5ab6d1317d6b28d6e, seq_no=45, attempt_no=0}) took 1028 ms. Trace:
I20260804 14:51:55.371042 27631 rpcz_store.cc:276] 0804 14:51:54.341938 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:54.342013 (+ 75us) service_pool.cc:227] Handling call
0804 14:51:55.370788 (+1028775us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:54.342406 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:54.342484 (+ 78us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:54.342487 (+ 3us) write_op.cc:435] Acquired schema lock
0804 14:51:54.342488 (+ 1us) tablet.cc:660] Decoding operations
0804 14:51:54.342546 (+ 58us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:54.342560 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:54.342562 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:54.342594 (+ 32us) tablet.cc:699] Row locks acquired
0804 14:51:54.342595 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:51:54.342632 (+ 37us) write_op.cc:270] Start()
0804 14:51:54.878828 (+536196us) write_op.cc:276] Timestamp: P: 1785855114342586 usec, L: 0
0804 14:51:54.878832 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:51:54.879195 (+ 363us) log.cc:842] Serialized 354 byte log entry
0804 14:51:55.370249 (+491054us) op_driver.cc:444] REPLICATION: finished
0804 14:51:55.370377 (+ 128us) write_op.cc:301] APPLY: starting
0804 14:51:55.370398 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0804 14:51:55.370431 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0804 14:51:55.370446 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:51:55.370507 (+ 61us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:51:55.370535 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:51:55.370539 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:51:55.370612 (+ 73us) log.cc:842] Serialized 22 byte log entry
0804 14:51:55.370696 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:55.370710 (+ 14us) write_op.cc:454] Released schema lock
0804 14:51:55.370715 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":58,"num_ops":1,"prepare.queue_time_us":29,"prepare.run_cpu_time_us":787,"prepare.run_wall_time_us":536875,"replication_time_us":491395}]]}
I20260804 14:51:55.372304 26755 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:55.372514 27593 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260804 14:51:55.372957 27336 scanner-internal.cc:458] Time spent opening tablet: real 0.881s user 0.001s sys 0.000s
W20260804 14:51:55.372957 27335 scanner-internal.cc:458] Time spent opening tablet: real 0.881s user 0.001s sys 0.000s
I20260804 14:51:55.990337 27580 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:55.990558 27580 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:55.990659 27199 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.24.6.130:33363 (request call id 5545) took 1479 ms. Trace:
I20260804 14:51:55.990748 27199 rpcz_store.cc:276] 0804 14:51:54.510893 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:54.510957 (+ 64us) service_pool.cc:227] Handling call
0804 14:51:54.511117 (+ 160us) raft_consensus.cc:1457] Updating replica for 2 ops
0804 14:51:54.511188 (+ 71us) raft_consensus.cc:1520] Early marking committed up to index 44
0804 14:51:54.511375 (+ 187us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0804 14:51:54.511496 (+ 121us) write_op.cc:270] Start()
0804 14:51:55.199688 (+688192us) write_op.cc:276] Timestamp: P: 1785855113512790 usec, L: 0
0804 14:51:55.200084 (+ 396us) write_op.cc:270] Start()
0804 14:51:55.989304 (+789220us) write_op.cc:276] Timestamp: P: 1785855114176831 usec, L: 0
0804 14:51:55.989911 (+ 607us) log.cc:842] Serialized 708 byte log entry
0804 14:51:55.990055 (+ 144us) raft_consensus.cc:1631] Marking committed up to 45
0804 14:51:55.990249 (+ 194us) raft_consensus.cc:1683] Filling consensus response to leader.
0804 14:51:55.990288 (+ 39us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0804 14:51:55.990575 (+ 287us) raft_consensus.cc:1671] finished
0804 14:51:55.990579 (+ 4us) raft_consensus.cc:1677] UpdateReplicas() finished
0804 14:51:55.990641 (+ 62us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:55.200087 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:55.200191 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:55.200195 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:51:55.200196 (+ 1us) tablet.cc:660] Decoding operations
0804 14:51:55.200271 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:55.200281 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:55.200283 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:55.200322 (+ 39us) tablet.cc:699] Row locks acquired
0804 14:51:55.200323 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:51:55.990129 (+789806us) op_driver.cc:444] REPLICATION: finished
0804 14:51:55.990371 (+ 242us) write_op.cc:301] APPLY: starting
0804 14:51:55.990389 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0804 14:51:55.990421 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0804 14:51:55.990422 (+ 1us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:51:55.990481 (+ 59us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:51:55.990511 (+ 30us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:51:55.990535 (+ 24us) write_op.cc:312] APPLY: finished
0804 14:51:55.990596 (+ 61us) log.cc:842] Serialized 22 byte log entry
0804 14:51:55.990650 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:55.990662 (+ 12us) write_op.cc:454] Released schema lock
0804 14:51:55.990667 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0804 14:51:55.990730 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:55.990733 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0804 14:51:55.989923 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:55.990109 (+ 186us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:55.990114 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:51:55.990116 (+ 2us) tablet.cc:660] Decoding operations
0804 14:51:55.990244 (+ 128us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:55.990256 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:55.990259 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:55.990326 (+ 67us) tablet.cc:699] Row locks acquired
0804 14:51:55.990328 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":86,"threads_started":1,"wal-append.queue_time_us":155,"child_traces":[["op",{"apply.queue_time_us":194,"apply.run_cpu_time_us":402,"apply.run_wall_time_us":402,"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":257,"prepare.run_cpu_time_us":314,"prepare.run_wall_time_us":314,"replication_time_us":1478617,"thread_start_us":167,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":278,"prepare.run_cpu_time_us":514,"prepare.run_wall_time_us":562,"thread_start_us":98,"threads_started":1}]]}
I20260804 14:51:55.991168 27580 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:55.991808 26845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:55.991870 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:55.992409 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:55.992640 27580 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:56.280077 27593 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:56.280220 27593 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:56.280766 27593 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:56.281266 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:56.281299 27197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:56.282073 26757 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:56.282338 27593 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:56.282593 27646 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50938 (ReqId={client: 2f11fb73142a45a5ab6d1317d6b28d6e, seq_no=46, attempt_no=0}) took 1175 ms. Trace:
I20260804 14:51:56.282655 27646 rpcz_store.cc:276] 0804 14:51:55.107064 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:55.107169 (+ 105us) service_pool.cc:227] Handling call
0804 14:51:56.282579 (+1175410us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:55.107676 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:55.107776 (+ 100us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:55.107779 (+ 3us) write_op.cc:435] Acquired schema lock
0804 14:51:55.107781 (+ 2us) tablet.cc:660] Decoding operations
0804 14:51:55.107848 (+ 67us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:55.107854 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:55.107857 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:55.107891 (+ 34us) tablet.cc:699] Row locks acquired
0804 14:51:55.107892 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:51:55.107922 (+ 30us) write_op.cc:270] Start()
0804 14:51:55.407068 (+299146us) write_op.cc:276] Timestamp: P: 1785855115107865 usec, L: 0
0804 14:51:55.407071 (+ 3us) op_driver.cc:331] REPLICATION: starting
0804 14:51:55.407261 (+ 190us) log.cc:842] Serialized 354 byte log entry
0804 14:51:56.281854 (+874593us) op_driver.cc:444] REPLICATION: finished
0804 14:51:56.282136 (+ 282us) write_op.cc:301] APPLY: starting
0804 14:51:56.282161 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
0804 14:51:56.282200 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0804 14:51:56.282216 (+ 16us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:51:56.282281 (+ 65us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:51:56.282314 (+ 33us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:51:56.282320 (+ 6us) write_op.cc:312] APPLY: finished
0804 14:51:56.282407 (+ 87us) log.cc:842] Serialized 22 byte log entry
0804 14:51:56.282483 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:56.282498 (+ 15us) write_op.cc:454] Released schema lock
0804 14:51:56.282512 (+ 14us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":200,"num_ops":1,"prepare.queue_time_us":58,"prepare.run_cpu_time_us":607,"prepare.run_wall_time_us":299681,"raft.queue_time_us":43,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":146,"replication_time_us":874762,"spinlock_wait_cycles":16128,"thread_start_us":76,"threads_started":1}]]}
I20260804 14:51:56.299815 27593 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:56.300069 27593 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:56.300668 27593 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:56.301049 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 49 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:56.301219 27197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 49 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:56.301647 26755 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:56.302029 27593 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260804 14:51:56.304004 27335 scanner-internal.cc:458] Time spent opening tablet: real 0.780s user 0.001s sys 0.000s
W20260804 14:51:56.304047 27336 scanner-internal.cc:458] Time spent opening tablet: real 0.780s user 0.001s sys 0.000s
I20260804 14:51:56.904258 27593 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:56.904423 27593 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:56.904985 27593 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:56.905504 27197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:56.905505 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:56.906020 26757 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:56.906221 27593 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:57.134057 27580 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:57.134214 27580 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:57.134409 27199 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.24.6.130:33363 (request call id 5546) took 1142 ms. Trace:
I20260804 14:51:57.134737 27580 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:57.134503 27199 rpcz_store.cc:276] 0804 14:51:55.992405 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:55.992478 (+ 73us) service_pool.cc:227] Handling call
0804 14:51:55.992667 (+ 189us) raft_consensus.cc:1457] Updating replica for 3 ops
0804 14:51:55.992825 (+ 158us) raft_consensus.cc:1520] Early marking committed up to index 46
0804 14:51:55.992902 (+ 77us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0804 14:51:55.993013 (+ 111us) write_op.cc:270] Start()
0804 14:51:56.010184 (+ 17171us) write_op.cc:276] Timestamp: P: 1785855114342586 usec, L: 0
0804 14:51:56.010472 (+ 288us) write_op.cc:270] Start()
0804 14:51:56.134686 (+124214us) write_op.cc:276] Timestamp: P: 1785855115107865 usec, L: 0
0804 14:51:56.134975 (+ 289us) write_op.cc:270] Start()
0804 14:51:57.133194 (+998219us) write_op.cc:276] Timestamp: P: 1785855115407715 usec, L: 0
0804 14:51:57.133643 (+ 449us) log.cc:842] Serialized 1062 byte log entry
0804 14:51:57.133810 (+ 167us) raft_consensus.cc:1631] Marking committed up to 47
0804 14:51:57.133979 (+ 169us) raft_consensus.cc:1683] Filling consensus response to leader.
0804 14:51:57.134015 (+ 36us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0804 14:51:57.134302 (+ 287us) raft_consensus.cc:1671] finished
0804 14:51:57.134304 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0804 14:51:57.134390 (+ 86us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:56.010507 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:56.010628 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:56.010632 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:51:56.010634 (+ 2us) tablet.cc:660] Decoding operations
0804 14:51:56.010713 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:56.010728 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:56.010731 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:56.010774 (+ 43us) tablet.cc:699] Row locks acquired
0804 14:51:56.010776 (+ 2us) write_op.cc:260] PREPARE: finished
0804 14:51:57.133845 (+1123069us) op_driver.cc:444] REPLICATION: finished
0804 14:51:57.134121 (+ 276us) write_op.cc:301] APPLY: starting
0804 14:51:57.134138 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0804 14:51:57.134171 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0804 14:51:57.134175 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:51:57.134236 (+ 61us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:51:57.134265 (+ 29us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:51:57.134285 (+ 20us) write_op.cc:312] APPLY: finished
0804 14:51:57.134349 (+ 64us) log.cc:842] Serialized 22 byte log entry
0804 14:51:57.134424 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:57.134438 (+ 14us) write_op.cc:454] Released schema lock
0804 14:51:57.134446 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0804 14:51:57.134510 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:57.134512 (+ 2us) write_op.cc:454] Released schema lock
Related trace 'op':
0804 14:51:56.134959 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:56.135074 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:56.135079 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:51:56.135081 (+ 2us) tablet.cc:660] Decoding operations
0804 14:51:56.135212 (+ 131us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:56.135260 (+ 48us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:56.135263 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:56.135313 (+ 50us) tablet.cc:699] Row locks acquired
0804 14:51:56.135314 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0804 14:51:57.133573 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:57.133676 (+ 103us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:57.133680 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:51:57.133681 (+ 1us) tablet.cc:660] Decoding operations
0804 14:51:57.133752 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:57.133762 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:57.133764 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:57.133898 (+ 134us) tablet.cc:699] Row locks acquired
0804 14:51:57.133899 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"spinlock_wait_cycles":149888,"thread_start_us":113,"threads_started":1,"wal-append.queue_time_us":196,"child_traces":[["op",{"apply.queue_time_us":226,"apply.run_cpu_time_us":432,"apply.run_wall_time_us":431,"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":105,"prepare.run_cpu_time_us":374,"prepare.run_wall_time_us":374,"replication_time_us":1140818,"thread_start_us":66,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":37,"prepare.queue_time_us":87,"prepare.run_cpu_time_us":450,"prepare.run_wall_time_us":451}],["op",{"num_ops":1,"partition_lock_wait_us":4,"prepare.queue_time_us":220,"prepare.run_cpu_time_us":390,"prepare.run_wall_time_us":390,"thread_start_us":85,"threads_started":1}]]}
I20260804 14:51:57.135298 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 49 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:57.135406 26844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 49 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:57.135845 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:57.136041 27580 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:57.161473 27580 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:57.161628 27580 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:57.162175 27580 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:57.162686 26845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:57.162748 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:57.163193 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:57.163440 27580 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:57.562815 27580 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:57.562956 27580 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:57.563616 27580 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:57.564096 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 51 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:57.564119 26844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 51 } } ignore_live_leader: false dest_uuid: "a2bfdd3e90e148468b3755c42cad8856" is_pre_election: true
I20260804 14:51:57.564549 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:57.564762 27667 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50938 (ReqId={client: 2f11fb73142a45a5ab6d1317d6b28d6e, seq_no=49, attempt_no=0}) took 1260 ms. Trace:
I20260804 14:51:57.564836 27580 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:57.564836 27667 rpcz_store.cc:276] 0804 14:51:56.303885 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:56.303959 (+ 74us) service_pool.cc:227] Handling call
0804 14:51:57.564747 (+1260788us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:56.507762 (+ 0us) write_op.cc:183] PREPARE: starting on tablet df07e91451bc49ba978359f97a315ebe
0804 14:51:56.507850 (+ 88us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:56.507854 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:51:56.507855 (+ 1us) tablet.cc:660] Decoding operations
0804 14:51:56.507949 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:56.507960 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:56.507963 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:56.508007 (+ 44us) tablet.cc:699] Row locks acquired
0804 14:51:56.508010 (+ 3us) write_op.cc:260] PREPARE: finished
0804 14:51:56.508053 (+ 43us) write_op.cc:270] Start()
0804 14:51:57.308187 (+800134us) write_op.cc:276] Timestamp: P: 1785855116508017 usec, L: 0
0804 14:51:57.308190 (+ 3us) op_driver.cc:331] REPLICATION: starting
0804 14:51:57.308347 (+ 157us) log.cc:842] Serialized 354 byte log entry
0804 14:51:57.563760 (+255413us) op_driver.cc:444] REPLICATION: finished
0804 14:51:57.564136 (+ 376us) write_op.cc:301] APPLY: starting
0804 14:51:57.564157 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0804 14:51:57.564190 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0804 14:51:57.564205 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:51:57.564325 (+ 120us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:51:57.564496 (+ 171us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:51:57.564502 (+ 6us) write_op.cc:312] APPLY: finished
0804 14:51:57.564583 (+ 81us) log.cc:842] Serialized 22 byte log entry
0804 14:51:57.564652 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:51:57.564668 (+ 16us) write_op.cc:454] Released schema lock
0804 14:51:57.564673 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":317,"num_ops":1,"prepare.queue_time_us":203437,"prepare.run_cpu_time_us":642,"prepare.run_wall_time_us":800698,"raft.queue_time_us":241,"raft.run_cpu_time_us":400,"raft.run_wall_time_us":423,"replication_time_us":255544,"spinlock_wait_cycles":16128,"thread_start_us":117,"threads_started":1}]]}
I20260804 14:51:57.636904 27174 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260804 14:51:57.638374 26994 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260804 14:51:57.764132 27593 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:57.764274 27593 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:57.764945 27593 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489), 59538ca2ea484ee197ada4dbcbeac3bf (127.24.6.131:42021)
I20260804 14:51:57.765447 27197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 51 } } ignore_live_leader: false dest_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" is_pre_election: true
I20260804 14:51:57.765489 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "a2bfdd3e90e148468b3755c42cad8856" candidate_term: 2 candidate_status { last_received { term: 1 index: 51 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
I20260804 14:51:57.766295 26757 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2bfdd3e90e148468b3755c42cad8856; no voters: 59538ca2ea484ee197ada4dbcbeac3bf, 9105c783d9e14fa3af410b705c5daebd
I20260804 14:51:57.766465 27593 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P a2bfdd3e90e148468b3755c42cad8856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:51:57.766842 26819 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
862a2eb595284226b154fa6290ec0ce2 | 127.24.6.190:43405 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.24.6.148:38557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
59538ca2ea484ee197ada4dbcbeac3bf | 127.24.6.131:42021 | HEALTHY | <none> | 0 | 0
9105c783d9e14fa3af410b705c5daebd | 127.24.6.130:37489 | HEALTHY | <none> | 1 | 0
a2bfdd3e90e148468b3755c42cad8856 | 127.24.6.129:42087 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
inject_latency_ms_before_starting_op | 1000 | hidden,advanced | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.24.6.129 | experimental | 127.24.6.129:42087
local_ip_for_outbound_sockets | 127.24.6.130 | experimental | 127.24.6.130:37489
local_ip_for_outbound_sockets | 127.24.6.131 | experimental | 127.24.6.131:42021
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.24.6.129:42087
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.24.6.130:37489
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.24.6.131:42021
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.24.6.148:38557 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260804 14:51:57.791733 24602 log_verifier.cc:126] Checking tablet df07e91451bc49ba978359f97a315ebe
I20260804 14:51:57.836759 24602 log_verifier.cc:177] Verified matching terms for 51 ops in tablet df07e91451bc49ba978359f97a315ebe
I20260804 14:51:57.864755 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 26736
I20260804 14:51:57.872893 27580 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:57.873073 27580 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:57.873597 27580 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
I20260804 14:51:57.874010 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df07e91451bc49ba978359f97a315ebe" candidate_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 51 } } ignore_live_leader: false dest_uuid: "9105c783d9e14fa3af410b705c5daebd" is_pre_election: true
W20260804 14:51:57.880654 27108 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.24.6.129:42087 timed out after 0.006s (SENT) [suppressed 1 similar messages]
W20260804 14:51:57.882915 27108 connection.cc:535] client connection to 127.24.6.129:42087 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260804 14:51:57.883081 27108 leader_election.cc:334] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260804 14:51:57.883149 27108 leader_election.cc:302] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59538ca2ea484ee197ada4dbcbeac3bf; no voters: 9105c783d9e14fa3af410b705c5daebd, a2bfdd3e90e148468b3755c42cad8856
I20260804 14:51:57.883358 27580 raft_consensus.cc:2726] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260804 14:51:57.883881 26929 connection.cc:535] client connection to 127.24.6.129:42087 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260804 14:51:57.883997 26929 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 1 similar messages]
I20260804 14:51:57.885026 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 26906
I20260804 14:51:57.892238 27580 raft_consensus.cc:482] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9105c783d9e14fa3af410b705c5daebd)
I20260804 14:51:57.892423 27580 raft_consensus.cc:504] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfdd3e90e148468b3755c42cad8856" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 42087 } } peers { permanent_uuid: "9105c783d9e14fa3af410b705c5daebd" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 37489 } } peers { permanent_uuid: "59538ca2ea484ee197ada4dbcbeac3bf" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42021 } }
I20260804 14:51:57.893003 27580 leader_election.cc:288] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087), 9105c783d9e14fa3af410b705c5daebd (127.24.6.130:37489)
W20260804 14:51:57.894353 27108 leader_election.cc:334] T df07e91451bc49ba978359f97a315ebe P 59538ca2ea484ee197ada4dbcbeac3bf [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a2bfdd3e90e148468b3755c42cad8856 (127.24.6.129:42087): Network error: Client connection negotiation failed: client connection to 127.24.6.129:42087: connect: Connection refused (error 111)
W20260804 14:51:57.899717 27108 connection.cc:535] client connection to 127.24.6.130:37489 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260804 14:51:57.900161 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 27083
I20260804 14:51:57.917259 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 26612
2026-08-04T14:51:57Z chronyd exiting
[ OK ] RaftConsensusElectionITest.ChurnyElections (32338 ms)
[ RUN ] RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency
I20260804 14:51:57.947670 24602 test_util.cc:268] Using random seed: -873663731
I20260804 14:51:57.948565 24602 ts_itest-base.cc:112] Starting cluster with:
I20260804 14:51:57.948616 24602 ts_itest-base.cc:113] --------------
I20260804 14:51:57.948627 24602 ts_itest-base.cc:114] 3 tablet servers
I20260804 14:51:57.948638 24602 ts_itest-base.cc:115] 3 replicas per TS
I20260804 14:51:57.948666 24602 ts_itest-base.cc:116] --------------
2026-08-04T14:51:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-04T14:51:57Z Disabled control of system clock
I20260804 14:51:57.960273 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.6.190:37795
--webserver_interface=127.24.6.190
--webserver_port=0
--builtin_ntp_servers=127.24.6.148:42301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.6.190:37795 with env {}
W20260804 14:51:58.086961 27735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:51:58.087354 27735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:51:58.087435 27735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:51:58.092026 27735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260804 14:51:58.092139 27735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:51:58.092173 27735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260804 14:51:58.092213 27735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260804 14:51:58.097450 27735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:42301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.6.190:37795
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.6.190:37795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.24.6.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.27735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:51:58.098730 27735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:51:58.100301 27735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:51:58.110898 27740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:51:58.111259 27741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:51:58.111239 27735 server_base.cc:1034] running on GCE node
W20260804 14:51:58.111194 27747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:51:58.112106 27735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:51:58.113333 27735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:51:58.114501 27735 hybrid_clock.cc:648] HybridClock initialized: now 1785855118114489 us; error 31 us; skew 500 ppm
I20260804 14:51:58.116643 27735 webserver.cc:460] Webserver started at http://127.24.6.190:35315/ using document root <none> and password file <none>
I20260804 14:51:58.117247 27735 fs_manager.cc:359] Metadata directory not provided
I20260804 14:51:58.117311 27735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:51:58.117491 27735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:51:58.119616 27735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "0a55c607a4694a67bf71a63db589c9d7"
format_stamp: "Formatted at 2026-08-04 14:51:58 on dist-test-slave-92m1"
I20260804 14:51:58.120262 27735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "0a55c607a4694a67bf71a63db589c9d7"
format_stamp: "Formatted at 2026-08-04 14:51:58 on dist-test-slave-92m1"
I20260804 14:51:58.124599 27735 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260804 14:51:58.127410 27754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:51:58.128895 27735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260804 14:51:58.129021 27735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
uuid: "0a55c607a4694a67bf71a63db589c9d7"
format_stamp: "Formatted at 2026-08-04 14:51:58 on dist-test-slave-92m1"
I20260804 14:51:58.129133 27735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:51:58.147333 27735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:51:58.148201 27735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:51:58.148370 27735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:51:58.156057 27735 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.190:37795
I20260804 14:51:58.156083 27827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.190:37795 every 8 connection(s)
I20260804 14:51:58.157353 27735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260804 14:51:58.158116 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 27735
I20260804 14:51:58.158272 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260804 14:51:58.162135 27830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:51:58.168207 27830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a55c607a4694a67bf71a63db589c9d7: Bootstrap starting.
I20260804 14:51:58.171018 27830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a55c607a4694a67bf71a63db589c9d7: Neither blocks nor log segments found. Creating new log.
I20260804 14:51:58.171838 27830 log.cc:824] T 00000000000000000000000000000000 P 0a55c607a4694a67bf71a63db589c9d7: Log is configured to *not* fsync() on all Append() calls
I20260804 14:51:58.173913 27830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a55c607a4694a67bf71a63db589c9d7: No bootstrap required, opened a new log
I20260804 14:51:58.178113 27830 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a55c607a4694a67bf71a63db589c9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a55c607a4694a67bf71a63db589c9d7" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 37795 } }
I20260804 14:51:58.178373 27830 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a55c607a4694a67bf71a63db589c9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:51:58.178438 27830 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a55c607a4694a67bf71a63db589c9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a55c607a4694a67bf71a63db589c9d7, State: Initialized, Role: FOLLOWER
I20260804 14:51:58.178900 27830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a55c607a4694a67bf71a63db589c9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a55c607a4694a67bf71a63db589c9d7" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 37795 } }
I20260804 14:51:58.179028 27830 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a55c607a4694a67bf71a63db589c9d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260804 14:51:58.179127 27830 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a55c607a4694a67bf71a63db589c9d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260804 14:51:58.179281 27830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a55c607a4694a67bf71a63db589c9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:51:58.180306 27830 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a55c607a4694a67bf71a63db589c9d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a55c607a4694a67bf71a63db589c9d7" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 37795 } }
I20260804 14:51:58.180696 27830 leader_election.cc:302] T 00000000000000000000000000000000 P 0a55c607a4694a67bf71a63db589c9d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a55c607a4694a67bf71a63db589c9d7; no voters:
I20260804 14:51:58.181107 27830 leader_election.cc:288] T 00000000000000000000000000000000 P 0a55c607a4694a67bf71a63db589c9d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260804 14:51:58.181262 27838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a55c607a4694a67bf71a63db589c9d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:51:58.181515 27838 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a55c607a4694a67bf71a63db589c9d7 [term 1 LEADER]: Becoming Leader. State: Replica: 0a55c607a4694a67bf71a63db589c9d7, State: Running, Role: LEADER
I20260804 14:51:58.181867 27838 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a55c607a4694a67bf71a63db589c9d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a55c607a4694a67bf71a63db589c9d7" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 37795 } }
I20260804 14:51:58.182590 27830 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a55c607a4694a67bf71a63db589c9d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260804 14:51:58.184010 27839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a55c607a4694a67bf71a63db589c9d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a55c607a4694a67bf71a63db589c9d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a55c607a4694a67bf71a63db589c9d7" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 37795 } } }
I20260804 14:51:58.184044 27841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a55c607a4694a67bf71a63db589c9d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a55c607a4694a67bf71a63db589c9d7. Latest consensus state: current_term: 1 leader_uuid: "0a55c607a4694a67bf71a63db589c9d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a55c607a4694a67bf71a63db589c9d7" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 37795 } } }
I20260804 14:51:58.184154 27839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a55c607a4694a67bf71a63db589c9d7 [sys.catalog]: This master's current role is: LEADER
I20260804 14:51:58.184158 27841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a55c607a4694a67bf71a63db589c9d7 [sys.catalog]: This master's current role is: LEADER
I20260804 14:51:58.184620 27855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260804 14:51:58.187726 27855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260804 14:51:58.193630 27855 catalog_manager.cc:1347] Generated new cluster ID: 557fddcef0d942a2a179b854123d0a4f
I20260804 14:51:58.193758 27855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260804 14:51:58.241263 27855 catalog_manager.cc:1370] Generated new certificate authority record
I20260804 14:51:58.242698 27855 catalog_manager.cc:1504] Loading token signing keys...
I20260804 14:51:58.250744 27855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a55c607a4694a67bf71a63db589c9d7: Generated new TSK 0
I20260804 14:51:58.251858 27855 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260804 14:51:58.265995 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.129:0
--local_ip_for_outbound_sockets=127.24.6.129
--webserver_interface=127.24.6.129
--webserver_port=0
--tserver_master_addrs=127.24.6.190:37795
--builtin_ntp_servers=127.24.6.148:42301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000
--consensus_inject_latency_ms_in_notifications=50 with env {}
W20260804 14:51:58.396951 27869 flags.cc:425] Enabled unsafe flag: --consensus_inject_latency_ms_in_notifications=50
W20260804 14:51:58.397276 27869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:51:58.397298 27869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:51:58.397336 27869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:51:58.401458 27869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:51:58.401579 27869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.129
I20260804 14:51:58.406708 27869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:42301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_inject_latency_ms_in_notifications=50
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.24.6.129
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.24.6.190:37795
--never_fsync=true
--heap_profile_path=/tmp/kudu.27869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.129
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:51:58.407945 27869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:51:58.409408 27869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:51:58.419950 27882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:51:58.419965 27878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:51:58.419961 27879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:51:58.420385 27869 server_base.cc:1034] running on GCE node
I20260804 14:51:58.420962 27869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:51:58.421684 27869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:51:58.422885 27869 hybrid_clock.cc:648] HybridClock initialized: now 1785855118422842 us; error 53 us; skew 500 ppm
I20260804 14:51:58.425222 27869 webserver.cc:460] Webserver started at http://127.24.6.129:40303/ using document root <none> and password file <none>
I20260804 14:51:58.425953 27869 fs_manager.cc:359] Metadata directory not provided
I20260804 14:51:58.426033 27869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:51:58.426335 27869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:51:58.428864 27869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "785631f53b2c4d8eafac62a9d2719f58"
format_stamp: "Formatted at 2026-08-04 14:51:58 on dist-test-slave-92m1"
I20260804 14:51:58.429602 27869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "785631f53b2c4d8eafac62a9d2719f58"
format_stamp: "Formatted at 2026-08-04 14:51:58 on dist-test-slave-92m1"
I20260804 14:51:58.434502 27869 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260804 14:51:58.437718 27890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:51:58.439446 27869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260804 14:51:58.439664 27869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "785631f53b2c4d8eafac62a9d2719f58"
format_stamp: "Formatted at 2026-08-04 14:51:58 on dist-test-slave-92m1"
I20260804 14:51:58.439805 27869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:51:58.465552 27869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:51:58.466359 27869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:51:58.466589 27869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:51:58.467360 27869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:51:58.468508 27869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:51:58.468560 27869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:51:58.468605 27869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:51:58.468622 27869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:51:58.478576 27869 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.129:43707
I20260804 14:51:58.478601 28051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.129:43707 every 8 connection(s)
I20260804 14:51:58.479743 27869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260804 14:51:58.485258 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 27869
I20260804 14:51:58.485374 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260804 14:51:58.487939 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.130:0
--local_ip_for_outbound_sockets=127.24.6.130
--webserver_interface=127.24.6.130
--webserver_port=0
--tserver_master_addrs=127.24.6.190:37795
--builtin_ntp_servers=127.24.6.148:42301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000
--consensus_inject_latency_ms_in_notifications=50 with env {}
I20260804 14:51:58.493049 28052 heartbeater.cc:347] Connected to a master server at 127.24.6.190:37795
I20260804 14:51:58.493407 28052 heartbeater.cc:464] Registering TS with master...
I20260804 14:51:58.494035 28052 heartbeater.cc:511] Master 127.24.6.190:37795 requested a full tablet report, sending...
I20260804 14:51:58.495941 27776 ts_manager.cc:194] Registered new tserver with Master: 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707)
I20260804 14:51:58.497241 27776 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.129:33357
W20260804 14:51:58.617256 28055 flags.cc:425] Enabled unsafe flag: --consensus_inject_latency_ms_in_notifications=50
W20260804 14:51:58.617573 28055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:51:58.617595 28055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:51:58.617687 28055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:51:58.622154 28055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:51:58.622293 28055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.130
I20260804 14:51:58.627905 28055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:42301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_inject_latency_ms_in_notifications=50
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.24.6.130
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.24.6.190:37795
--never_fsync=true
--heap_profile_path=/tmp/kudu.28055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.130
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:51:58.629233 28055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:51:58.630818 28055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:51:58.642313 28066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:51:58.642302 28061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:51:58.642302 28062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:51:58.642768 28055 server_base.cc:1034] running on GCE node
I20260804 14:51:58.643388 28055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:51:58.644236 28055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:51:58.645422 28055 hybrid_clock.cc:648] HybridClock initialized: now 1785855118645377 us; error 59 us; skew 500 ppm
I20260804 14:51:58.647663 28055 webserver.cc:460] Webserver started at http://127.24.6.130:41171/ using document root <none> and password file <none>
I20260804 14:51:58.648284 28055 fs_manager.cc:359] Metadata directory not provided
I20260804 14:51:58.648387 28055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:51:58.648674 28055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:51:58.650787 28055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "92ba21101c0a4918be8e047240f43c8b"
format_stamp: "Formatted at 2026-08-04 14:51:58 on dist-test-slave-92m1"
I20260804 14:51:58.651398 28055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "92ba21101c0a4918be8e047240f43c8b"
format_stamp: "Formatted at 2026-08-04 14:51:58 on dist-test-slave-92m1"
I20260804 14:51:58.655671 28055 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260804 14:51:58.658324 28075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:51:58.659971 28055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260804 14:51:58.660145 28055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "92ba21101c0a4918be8e047240f43c8b"
format_stamp: "Formatted at 2026-08-04 14:51:58 on dist-test-slave-92m1"
I20260804 14:51:58.660269 28055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:51:58.671679 28055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:51:58.672511 28055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:51:58.672802 28055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:51:58.673486 28055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:51:58.674561 28055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:51:58.674612 28055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:51:58.674656 28055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:51:58.674672 28055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:51:58.685144 28055 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.130:43373
I20260804 14:51:58.685216 28230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.130:43373 every 8 connection(s)
I20260804 14:51:58.686196 28055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260804 14:51:58.687472 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 28055
I20260804 14:51:58.687600 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260804 14:51:58.690371 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.131:0
--local_ip_for_outbound_sockets=127.24.6.131
--webserver_interface=127.24.6.131
--webserver_port=0
--tserver_master_addrs=127.24.6.190:37795
--builtin_ntp_servers=127.24.6.148:42301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000
--consensus_inject_latency_ms_in_notifications=50 with env {}
I20260804 14:51:58.700277 28231 heartbeater.cc:347] Connected to a master server at 127.24.6.190:37795
I20260804 14:51:58.700582 28231 heartbeater.cc:464] Registering TS with master...
I20260804 14:51:58.701197 28231 heartbeater.cc:511] Master 127.24.6.190:37795 requested a full tablet report, sending...
I20260804 14:51:58.702628 27776 ts_manager.cc:194] Registered new tserver with Master: 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:51:58.703644 27776 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.130:53675
W20260804 14:51:58.818449 28234 flags.cc:425] Enabled unsafe flag: --consensus_inject_latency_ms_in_notifications=50
W20260804 14:51:58.818770 28234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:51:58.818794 28234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:51:58.818830 28234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:51:58.823170 28234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:51:58.823390 28234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.131
I20260804 14:51:58.828644 28234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:42301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_inject_latency_ms_in_notifications=50
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.24.6.131
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.24.6.190:37795
--never_fsync=true
--heap_profile_path=/tmp/kudu.28234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.131
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:51:58.829855 28234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:51:58.831421 28234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:51:58.843408 28240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:51:58.843570 28241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:51:58.843634 28243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:51:58.843743 28234 server_base.cc:1034] running on GCE node
I20260804 14:51:58.844343 28234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:51:58.845114 28234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:51:58.846331 28234 hybrid_clock.cc:648] HybridClock initialized: now 1785855118846299 us; error 47 us; skew 500 ppm
I20260804 14:51:58.849043 28234 webserver.cc:460] Webserver started at http://127.24.6.131:44525/ using document root <none> and password file <none>
I20260804 14:51:58.849882 28234 fs_manager.cc:359] Metadata directory not provided
I20260804 14:51:58.850004 28234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:51:58.850284 28234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:51:58.852988 28234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "154119e307724bc095278d7e46a3eb05"
format_stamp: "Formatted at 2026-08-04 14:51:58 on dist-test-slave-92m1"
I20260804 14:51:58.853540 28234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "154119e307724bc095278d7e46a3eb05"
format_stamp: "Formatted at 2026-08-04 14:51:58 on dist-test-slave-92m1"
I20260804 14:51:58.857820 28234 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260804 14:51:58.860783 28251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:51:58.862498 28234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260804 14:51:58.862623 28234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "154119e307724bc095278d7e46a3eb05"
format_stamp: "Formatted at 2026-08-04 14:51:58 on dist-test-slave-92m1"
I20260804 14:51:58.862732 28234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:51:58.876778 28234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:51:58.877614 28234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:51:58.877805 28234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:51:58.878470 28234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:51:58.879659 28234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:51:58.879715 28234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:51:58.879762 28234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:51:58.879787 28234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:51:58.890112 28234 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.131:42591
I20260804 14:51:58.890146 28406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.131:42591 every 8 connection(s)
I20260804 14:51:58.891386 28234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260804 14:51:58.900061 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 28234
I20260804 14:51:58.900207 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260804 14:51:58.903378 28407 heartbeater.cc:347] Connected to a master server at 127.24.6.190:37795
I20260804 14:51:58.903673 28407 heartbeater.cc:464] Registering TS with master...
I20260804 14:51:58.904266 28407 heartbeater.cc:511] Master 127.24.6.190:37795 requested a full tablet report, sending...
I20260804 14:51:58.905496 27775 ts_manager.cc:194] Registered new tserver with Master: 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:51:58.906177 27775 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.131:46969
I20260804 14:51:58.916049 24602 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260804 14:51:58.918032 24602 test_util.cc:268] Using random seed: -872693366
I20260804 14:51:58.931603 27775 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45350:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260804 14:51:58.933173 27775 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260804 14:51:58.954097 28135 tablet_service.cc:1467] Processing CreateTablet for tablet 6cd92ff096234dafa214acc72d617f73 (DEFAULT_TABLE table=test-workload [id=41255276b38541d68bbfe0d0057a0118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:51:58.954260 27955 tablet_service.cc:1467] Processing CreateTablet for tablet 6cd92ff096234dafa214acc72d617f73 (DEFAULT_TABLE table=test-workload [id=41255276b38541d68bbfe0d0057a0118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:51:58.954262 28316 tablet_service.cc:1467] Processing CreateTablet for tablet 6cd92ff096234dafa214acc72d617f73 (DEFAULT_TABLE table=test-workload [id=41255276b38541d68bbfe0d0057a0118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:51:58.955365 28135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cd92ff096234dafa214acc72d617f73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:51:58.955430 27955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cd92ff096234dafa214acc72d617f73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:51:58.955430 28316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cd92ff096234dafa214acc72d617f73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:51:58.962044 28430 tablet_bootstrap.cc:492] T 6cd92ff096234dafa214acc72d617f73 P 92ba21101c0a4918be8e047240f43c8b: Bootstrap starting.
I20260804 14:51:58.963977 28432 tablet_bootstrap.cc:492] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05: Bootstrap starting.
I20260804 14:51:58.964717 28430 tablet_bootstrap.cc:654] T 6cd92ff096234dafa214acc72d617f73 P 92ba21101c0a4918be8e047240f43c8b: Neither blocks nor log segments found. Creating new log.
I20260804 14:51:58.965591 28430 log.cc:824] T 6cd92ff096234dafa214acc72d617f73 P 92ba21101c0a4918be8e047240f43c8b: Log is configured to *not* fsync() on all Append() calls
I20260804 14:51:58.966399 28432 tablet_bootstrap.cc:654] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05: Neither blocks nor log segments found. Creating new log.
I20260804 14:51:58.966495 28433 tablet_bootstrap.cc:492] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58: Bootstrap starting.
I20260804 14:51:58.967423 28432 log.cc:824] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05: Log is configured to *not* fsync() on all Append() calls
I20260804 14:51:58.968228 28430 tablet_bootstrap.cc:492] T 6cd92ff096234dafa214acc72d617f73 P 92ba21101c0a4918be8e047240f43c8b: No bootstrap required, opened a new log
I20260804 14:51:58.968521 28430 ts_tablet_manager.cc:1397] T 6cd92ff096234dafa214acc72d617f73 P 92ba21101c0a4918be8e047240f43c8b: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260804 14:51:58.969094 28433 tablet_bootstrap.cc:654] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58: Neither blocks nor log segments found. Creating new log.
I20260804 14:51:58.969790 28432 tablet_bootstrap.cc:492] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05: No bootstrap required, opened a new log
I20260804 14:51:58.969820 28433 log.cc:824] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58: Log is configured to *not* fsync() on all Append() calls
I20260804 14:51:58.970027 28432 ts_tablet_manager.cc:1397] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260804 14:51:58.971664 28433 tablet_bootstrap.cc:492] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58: No bootstrap required, opened a new log
I20260804 14:51:58.971879 28433 ts_tablet_manager.cc:1397] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260804 14:51:58.974085 28430 raft_consensus.cc:348] T 6cd92ff096234dafa214acc72d617f73 P 92ba21101c0a4918be8e047240f43c8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:51:58.974438 28430 raft_consensus.cc:374] T 6cd92ff096234dafa214acc72d617f73 P 92ba21101c0a4918be8e047240f43c8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:51:58.974514 28430 raft_consensus.cc:729] T 6cd92ff096234dafa214acc72d617f73 P 92ba21101c0a4918be8e047240f43c8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ba21101c0a4918be8e047240f43c8b, State: Initialized, Role: FOLLOWER
I20260804 14:51:58.975101 28430 consensus_queue.cc:260] T 6cd92ff096234dafa214acc72d617f73 P 92ba21101c0a4918be8e047240f43c8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:51:58.975176 28432 raft_consensus.cc:348] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:51:58.975482 28432 raft_consensus.cc:374] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:51:58.975559 28432 raft_consensus.cc:729] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 154119e307724bc095278d7e46a3eb05, State: Initialized, Role: FOLLOWER
I20260804 14:51:58.975854 28433 raft_consensus.cc:348] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:51:58.976081 28231 heartbeater.cc:503] Master 127.24.6.190:37795 was elected leader, sending a full tablet report...
I20260804 14:51:58.976127 28433 raft_consensus.cc:374] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:51:58.976177 28433 raft_consensus.cc:729] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 785631f53b2c4d8eafac62a9d2719f58, State: Initialized, Role: FOLLOWER
I20260804 14:51:58.976094 28432 consensus_queue.cc:260] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:51:58.976650 28433 consensus_queue.cc:260] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:51:58.976786 28430 ts_tablet_manager.cc:1428] T 6cd92ff096234dafa214acc72d617f73 P 92ba21101c0a4918be8e047240f43c8b: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260804 14:51:58.977159 28439 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 92ba21101c0a4918be8e047240f43c8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260804 14:51:58.977316 28439 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 92ba21101c0a4918be8e047240f43c8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:51:58.977439 28432 ts_tablet_manager.cc:1428] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260804 14:51:58.977548 28433 ts_tablet_manager.cc:1428] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260804 14:51:58.977794 28407 heartbeater.cc:503] Master 127.24.6.190:37795 was elected leader, sending a full tablet report...
I20260804 14:51:58.977890 28052 heartbeater.cc:503] Master 127.24.6.190:37795 was elected leader, sending a full tablet report...
I20260804 14:51:58.978235 28444 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260804 14:51:58.978425 28444 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:51:58.980113 28444 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:51:58.980509 28441 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260804 14:51:58.980752 28441 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
W20260804 14:51:58.981019 28053 tablet.cc:2369] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260804 14:51:58.983745 28441 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:51:58.984393 28439 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 92ba21101c0a4918be8e047240f43c8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
W20260804 14:51:58.986992 28253 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.24.6.129:43707 timed out after 0.007s (ON_OUTBOUND_QUEUE)
I20260804 14:51:58.990044 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:51:58.990348 28347 raft_consensus.cc:2454] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 785631f53b2c4d8eafac62a9d2719f58 in term 0.
W20260804 14:51:58.990677 27891 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.24.6.130:43373 for RPC RequestConsensusVote timed out after 0.008s (ON_OUTBOUND_QUEUE)
W20260804 14:51:58.991541 28077 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.24.6.129:43707 timed out after 0.008s (SENT)
I20260804 14:51:58.992182 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "92ba21101c0a4918be8e047240f43c8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:51:58.992255 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:51:58.992367 28347 raft_consensus.cc:2454] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92ba21101c0a4918be8e047240f43c8b in term 0.
I20260804 14:51:58.993567 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "92ba21101c0a4918be8e047240f43c8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:51:58.993726 28344 raft_consensus.cc:2454] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92ba21101c0a4918be8e047240f43c8b in term 0.
I20260804 14:51:58.994251 27988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "92ba21101c0a4918be8e047240f43c8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:51:58.994572 27988 raft_consensus.cc:2454] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92ba21101c0a4918be8e047240f43c8b in term 0.
I20260804 14:51:58.994812 28078 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 92ba21101c0a4918be8e047240f43c8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b; no voters:
I20260804 14:51:58.994251 27989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "92ba21101c0a4918be8e047240f43c8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:51:58.995128 28439 raft_consensus.cc:2781] T 6cd92ff096234dafa214acc72d617f73 P 92ba21101c0a4918be8e047240f43c8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260804 14:51:58.995253 28439 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 92ba21101c0a4918be8e047240f43c8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260804 14:51:58.995214 27989 raft_consensus.cc:2454] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92ba21101c0a4918be8e047240f43c8b in term 0.
I20260804 14:51:58.995313 28439 raft_consensus.cc:3037] T 6cd92ff096234dafa214acc72d617f73 P 92ba21101c0a4918be8e047240f43c8b [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:51:58.996544 27989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:51:58.996686 27989 raft_consensus.cc:2454] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 154119e307724bc095278d7e46a3eb05 in term 0.
I20260804 14:51:58.996629 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:51:58.996634 28165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:51:58.997030 28253 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 154119e307724bc095278d7e46a3eb05, 785631f53b2c4d8eafac62a9d2719f58; no voters:
I20260804 14:51:58.997112 28439 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 92ba21101c0a4918be8e047240f43c8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:51:58.997350 28166 raft_consensus.cc:2379] T 6cd92ff096234dafa214acc72d617f73 P 92ba21101c0a4918be8e047240f43c8b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 785631f53b2c4d8eafac62a9d2719f58 in current term 1: Already voted for candidate 92ba21101c0a4918be8e047240f43c8b in this term.
I20260804 14:51:58.997506 28439 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 92ba21101c0a4918be8e047240f43c8b [CANDIDATE]: Term 1 election: Requested vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:51:58.997568 28440 raft_consensus.cc:2781] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260804 14:51:58.997696 28440 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260804 14:51:58.997756 28440 raft_consensus.cc:3037] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:51:58.997750 27891 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:51:58.998011 28441 raft_consensus.cc:3037] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:51:58.998528 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "92ba21101c0a4918be8e047240f43c8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05"
I20260804 14:51:58.998821 27989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "92ba21101c0a4918be8e047240f43c8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58"
I20260804 14:51:58.999011 28440 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:51:58.999245 28441 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260804 14:51:58.999289 28344 raft_consensus.cc:2379] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 92ba21101c0a4918be8e047240f43c8b in current term 1: Already voted for candidate 154119e307724bc095278d7e46a3eb05 in this term.
I20260804 14:51:58.999490 28440 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 1 election: Requested vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:51:58.999897 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b"
I20260804 14:51:59.000083 28166 raft_consensus.cc:2379] T 6cd92ff096234dafa214acc72d617f73 P 92ba21101c0a4918be8e047240f43c8b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 154119e307724bc095278d7e46a3eb05 in current term 1: Already voted for candidate 92ba21101c0a4918be8e047240f43c8b in this term.
I20260804 14:51:59.000100 27988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58"
I20260804 14:51:59.000933 27989 raft_consensus.cc:2454] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92ba21101c0a4918be8e047240f43c8b in term 1.
I20260804 14:51:59.001809 28077 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 92ba21101c0a4918be8e047240f43c8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b; no voters: 154119e307724bc095278d7e46a3eb05
I20260804 14:51:59.001804 28252 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:51:59.002070 28440 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260804 14:51:59.002094 28439 raft_consensus.cc:2781] T 6cd92ff096234dafa214acc72d617f73 P 92ba21101c0a4918be8e047240f43c8b [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:51:59.002228 28439 raft_consensus.cc:686] T 6cd92ff096234dafa214acc72d617f73 P 92ba21101c0a4918be8e047240f43c8b [term 1 LEADER]: Becoming Leader. State: Replica: 92ba21101c0a4918be8e047240f43c8b, State: Running, Role: LEADER
I20260804 14:51:59.002583 28439 consensus_queue.cc:237] T 6cd92ff096234dafa214acc72d617f73 P 92ba21101c0a4918be8e047240f43c8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:51:59.006239 28458 consensus_queue.cc:1035] T 6cd92ff096234dafa214acc72d617f73 P 92ba21101c0a4918be8e047240f43c8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260804 14:51:59.006506 28439 consensus_queue.cc:1035] T 6cd92ff096234dafa214acc72d617f73 P 92ba21101c0a4918be8e047240f43c8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260804 14:51:59.006357 27776 catalog_manager.cc:5495] T 6cd92ff096234dafa214acc72d617f73 P 92ba21101c0a4918be8e047240f43c8b reported cstate change: term changed from 0 to 1, leader changed from <none> to 92ba21101c0a4918be8e047240f43c8b (127.24.6.130). New cstate: current_term: 1 leader_uuid: "92ba21101c0a4918be8e047240f43c8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } health_report { overall_health: UNKNOWN } } }
I20260804 14:51:59.009126 28441 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260804 14:51:59.009248 28441 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:51:59.009653 28441 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:51:59.010198 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:51:59.010443 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:51:59.010823 27893 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:51:59.011073 28441 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260804 14:51:59.012187 28438 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260804 14:51:59.012244 28438 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20260804 14:51:59.180016 28109 tablet_service.cc:2908] Rejecting scan request for tablet 6cd92ff096234dafa214acc72d617f73: Uninitialized: clean time has not yet been initialized
W20260804 14:51:59.180269 28108 tablet_service.cc:2908] Rejecting scan request for tablet 6cd92ff096234dafa214acc72d617f73: Uninitialized: clean time has not yet been initialized
W20260804 14:51:59.187597 28232 tablet.cc:2369] T 6cd92ff096234dafa214acc72d617f73 P 92ba21101c0a4918be8e047240f43c8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260804 14:51:59.189318 28283 tablet_service.cc:2970] error setting up scanner 28a06d86461349979f718e635473cb19 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.148s (maximum allowed 0.100s): new_scan_request { tablet_id: "6cd92ff096234dafa214acc72d617f73" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\273\373\307\323\006\"3\n\005slave\022*\n 41255276b38541d68bbfe0d0057a0118\020\001\030\001 \001(\001" signature: "<redacted>"")\215P\256\002\302\361\025)\236\rI]\255\241^\250\030+\004\3636!j\276lc$\025\300l\027N\275O\035T\274\302\3555\200\216\322\212\242\210\302[\342r\211s\325PR\200\310M\227\201x\204\354" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "a778d080873b4800b57ddf4a1fab5db6"
W20260804 14:51:59.189313 28285 tablet_service.cc:2970] error setting up scanner 1713664fa3cd45bb8d504647b707e07e with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.148s (maximum allowed 0.100s): new_scan_request { tablet_id: "6cd92ff096234dafa214acc72d617f73" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\273\373\307\323\006\"3\n\005slave\022*\n 41255276b38541d68bbfe0d0057a0118\020\001\030\001 \001(\001" signature: "<redacted>"")\215P\256\002\302\361\025)\236\rI]\255\241^\250\030+\004\3636!j\276lc$\025\300l\027N\275O\035T\274\302\3555\200\216\322\212\242\210\302[\342r\211s\325PR\200\310M\227\201x\204\354" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "f38c4acc68bb473786d01674d0ae9341"
W20260804 14:51:59.212369 28283 tablet_service.cc:2970] error setting up scanner 083c318baa37429a95a726b6e5b58547 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.171s (maximum allowed 0.100s): new_scan_request { tablet_id: "6cd92ff096234dafa214acc72d617f73" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\273\373\307\323\006\"3\n\005slave\022*\n 41255276b38541d68bbfe0d0057a0118\020\001\030\001 \001(\001" signature: "<redacted>"")\215P\256\002\302\361\025)\236\rI]\255\241^\250\030+\004\3636!j\276lc$\025\300l\027N\275O\035T\274\302\3555\200\216\322\212\242\210\302[\342r\211s\325PR\200\310M\227\201x\204\354" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "a778d080873b4800b57ddf4a1fab5db6"
W20260804 14:51:59.213925 28283 tablet_service.cc:2970] error setting up scanner 4e76f903d3fb4540a37b6948860917b8 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.172s (maximum allowed 0.100s): new_scan_request { tablet_id: "6cd92ff096234dafa214acc72d617f73" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\273\373\307\323\006\"3\n\005slave\022*\n 41255276b38541d68bbfe0d0057a0118\020\001\030\001 \001(\001" signature: "<redacted>"")\215P\256\002\302\361\025)\236\rI]\255\241^\250\030+\004\3636!j\276lc$\025\300l\027N\275O\035T\274\302\3555\200\216\322\212\242\210\302[\342r\211s\325PR\200\310M\227\201x\204\354" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "f38c4acc68bb473786d01674d0ae9341"
W20260804 14:51:59.278439 28283 tablet_service.cc:2970] error setting up scanner c974604870744d7fb5482dde449be7cc with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.237s (maximum allowed 0.100s): new_scan_request { tablet_id: "6cd92ff096234dafa214acc72d617f73" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\273\373\307\323\006\"3\n\005slave\022*\n 41255276b38541d68bbfe0d0057a0118\020\001\030\001 \001(\001" signature: "<redacted>"")\215P\256\002\302\361\025)\236\rI]\255\241^\250\030+\004\3636!j\276lc$\025\300l\027N\275O\035T\274\302\3555\200\216\322\212\242\210\302[\342r\211s\325PR\200\310M\227\201x\204\354" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "a778d080873b4800b57ddf4a1fab5db6"
W20260804 14:51:59.283926 28283 tablet_service.cc:2970] error setting up scanner a8addcb98f8e4f849091856fdc2d6fe0 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.242s (maximum allowed 0.100s): new_scan_request { tablet_id: "6cd92ff096234dafa214acc72d617f73" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\273\373\307\323\006\"3\n\005slave\022*\n 41255276b38541d68bbfe0d0057a0118\020\001\030\001 \001(\001" signature: "<redacted>"")\215P\256\002\302\361\025)\236\rI]\255\241^\250\030+\004\3636!j\276lc$\025\300l\027N\275O\035T\274\302\3555\200\216\322\212\242\210\302[\342r\211s\325PR\200\310M\227\201x\204\354" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "f38c4acc68bb473786d01674d0ae9341"
W20260804 14:51:59.360574 28283 tablet_service.cc:2970] error setting up scanner 2530286ad1c84851ab597420f3d1926e with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.319s (maximum allowed 0.100s): new_scan_request { tablet_id: "6cd92ff096234dafa214acc72d617f73" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\273\373\307\323\006\"3\n\005slave\022*\n 41255276b38541d68bbfe0d0057a0118\020\001\030\001 \001(\001" signature: "<redacted>"")\215P\256\002\302\361\025)\236\rI]\255\241^\250\030+\004\3636!j\276lc$\025\300l\027N\275O\035T\274\302\3555\200\216\322\212\242\210\302[\342r\211s\325PR\200\310M\227\201x\204\354" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "a778d080873b4800b57ddf4a1fab5db6"
W20260804 14:51:59.429828 28283 tablet_service.cc:2970] error setting up scanner 54a1aff466f94fd984786ba4c041570d with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.388s (maximum allowed 0.100s): new_scan_request { tablet_id: "6cd92ff096234dafa214acc72d617f73" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\273\373\307\323\006\"3\n\005slave\022*\n 41255276b38541d68bbfe0d0057a0118\020\001\030\001 \001(\001" signature: "<redacted>"")\215P\256\002\302\361\025)\236\rI]\255\241^\250\030+\004\3636!j\276lc$\025\300l\027N\275O\035T\274\302\3555\200\216\322\212\242\210\302[\342r\211s\325PR\200\310M\227\201x\204\354" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "f38c4acc68bb473786d01674d0ae9341"
W20260804 14:51:59.519930 28344 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260804 14:51:59.519958 27989 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260804 14:51:59.520020 28344 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20260804 14:51:59.520041 27989 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260804 14:51:59.527937 28474 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:51:59.528118 28474 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:51:59.528891 28474 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:51:59.529353 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:51:59.529454 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:51:59.530043 27891 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:51:59.530505 28474 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260804 14:51:59.618912 28283 tablet_service.cc:2970] error setting up scanner 2d629e49c0aa4322bffc1bd95f9dc0a6 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.577s (maximum allowed 0.100s): new_scan_request { tablet_id: "6cd92ff096234dafa214acc72d617f73" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\273\373\307\323\006\"3\n\005slave\022*\n 41255276b38541d68bbfe0d0057a0118\020\001\030\001 \001(\001" signature: "<redacted>"")\215P\256\002\302\361\025)\236\rI]\255\241^\250\030+\004\3636!j\276lc$\025\300l\027N\275O\035T\274\302\3555\200\216\322\212\242\210\302[\342r\211s\325PR\200\310M\227\201x\204\354" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "a778d080873b4800b57ddf4a1fab5db6"
W20260804 14:51:59.703912 28283 tablet_service.cc:2970] error setting up scanner 160d54d5b3f24aab98202d7c5b1757bf with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.662s (maximum allowed 0.100s): new_scan_request { tablet_id: "6cd92ff096234dafa214acc72d617f73" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\273\373\307\323\006\"3\n\005slave\022*\n 41255276b38541d68bbfe0d0057a0118\020\001\030\001 \001(\001" signature: "<redacted>"")\215P\256\002\302\361\025)\236\rI]\255\241^\250\030+\004\3636!j\276lc$\025\300l\027N\275O\035T\274\302\3555\200\216\322\212\242\210\302[\342r\211s\325PR\200\310M\227\201x\204\354" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "f38c4acc68bb473786d01674d0ae9341"
W20260804 14:51:59.973332 28283 tablet_service.cc:2970] error setting up scanner d8eefd27776a434eb2562a923194cc0a with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.932s (maximum allowed 0.100s): new_scan_request { tablet_id: "6cd92ff096234dafa214acc72d617f73" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\273\373\307\323\006\"3\n\005slave\022*\n 41255276b38541d68bbfe0d0057a0118\020\001\030\001 \001(\001" signature: "<redacted>"")\215P\256\002\302\361\025)\236\rI]\255\241^\250\030+\004\3636!j\276lc$\025\300l\027N\275O\035T\274\302\3555\200\216\322\212\242\210\302[\342r\211s\325PR\200\310M\227\201x\204\354" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "a778d080873b4800b57ddf4a1fab5db6"
I20260804 14:51:59.983376 28473 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:51:59.983525 28473 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:51:59.984537 28473 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:51:59.984802 27989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:51:59.985473 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:51:59.985958 28252 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:51:59.986207 28473 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:00.171077 28470 mvcc.cc:205] Tried to move back new op lower bound from 7314862567592280064 to 7314862567437864960. Current Snapshot: MvccSnapshot[applied={T|T < 7314862567592280064 or (T in {7314862567592280064})}]
W20260804 14:52:00.250197 28283 tablet_service.cc:2970] error setting up scanner c41cf4d1411a43389853f39ba43e82a7 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.265s (maximum allowed 0.100s): new_scan_request { tablet_id: "6cd92ff096234dafa214acc72d617f73" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\273\373\307\323\006\"3\n\005slave\022*\n 41255276b38541d68bbfe0d0057a0118\020\001\030\001 \001(\001" signature: "<redacted>"")\215P\256\002\302\361\025)\236\rI]\255\241^\250\030+\004\3636!j\276lc$\025\300l\027N\275O\035T\274\302\3555\200\216\322\212\242\210\302[\342r\211s\325PR\200\310M\227\201x\204\354" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "f38c4acc68bb473786d01674d0ae9341"
I20260804 14:52:00.419849 28481 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:00.420049 28481 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:00.421028 28481 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:52:00.421409 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:00.421522 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:52:00.421973 27893 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:00.422235 28481 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:00.461483 28484 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=0, attempt_no=0}) took 1425 ms. Trace:
I20260804 14:52:00.461602 28484 rpcz_store.cc:276] 0804 14:51:59.035899 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:59.035971 (+ 72us) service_pool.cc:227] Handling call
0804 14:52:00.461454 (+1425483us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:51:59.518207 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:51:59.518316 (+ 109us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:51:59.518321 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:51:59.518323 (+ 2us) tablet.cc:660] Decoding operations
0804 14:51:59.518431 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:51:59.518471 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0804 14:51:59.518474 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:51:59.518527 (+ 53us) tablet.cc:699] Row locks acquired
0804 14:51:59.518529 (+ 2us) write_op.cc:260] PREPARE: finished
0804 14:51:59.518577 (+ 48us) write_op.cc:270] Start()
0804 14:52:00.170749 (+652172us) write_op.cc:276] Timestamp: P: 1785855119518530 usec, L: 0
0804 14:52:00.170753 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:52:00.170921 (+ 168us) log.cc:842] Serialized 345 byte log entry
0804 14:52:00.460731 (+289810us) op_driver.cc:444] REPLICATION: finished
0804 14:52:00.461004 (+ 273us) write_op.cc:301] APPLY: starting
0804 14:52:00.461022 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:00.461054 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:00.461067 (+ 13us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:00.461163 (+ 96us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:00.461191 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:00.461194 (+ 3us) write_op.cc:312] APPLY: finished
0804 14:52:00.461263 (+ 69us) log.cc:842] Serialized 22 byte log entry
0804 14:52:00.461347 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:00.461362 (+ 15us) write_op.cc:454] Released schema lock
0804 14:52:00.461368 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":2080512,"child_traces":[["op",{"apply.queue_time_us":197,"mutex_wait_us":51,"num_ops":1,"prepare.queue_time_us":479390,"prepare.run_cpu_time_us":756,"prepare.run_wall_time_us":652833,"raft.queue_time_us":164,"raft.run_cpu_time_us":523,"raft.run_wall_time_us":551,"replication_time_us":289938,"spinlock_wait_cycles":370944,"thread_start_us":106,"threads_started":1}]]}
W20260804 14:52:00.807592 28283 tablet_service.cc:2970] error setting up scanner 1e417b9c98404eaaaf276008c83f8513 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.823s (maximum allowed 0.100s): new_scan_request { tablet_id: "6cd92ff096234dafa214acc72d617f73" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\273\373\307\323\006\"3\n\005slave\022*\n 41255276b38541d68bbfe0d0057a0118\020\001\030\001 \001(\001" signature: "<redacted>"")\215P\256\002\302\361\025)\236\rI]\255\241^\250\030+\004\3636!j\276lc$\025\300l\027N\275O\035T\274\302\3555\200\216\322\212\242\210\302[\342r\211s\325PR\200\310M\227\201x\204\354" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "a778d080873b4800b57ddf4a1fab5db6"
I20260804 14:52:01.031193 28473 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:01.031381 28473 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:01.032049 28473 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:01.032546 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:01.032590 27988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:01.033170 28252 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
W20260804 14:52:01.468782 28283 tablet_service.cc:2970] error setting up scanner 188051ce7ff740389c978675d048a4bc with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.435s (maximum allowed 0.100s): new_scan_request { tablet_id: "6cd92ff096234dafa214acc72d617f73" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\273\373\307\323\006\"3\n\005slave\022*\n 41255276b38541d68bbfe0d0057a0118\020\001\030\001 \001(\001" signature: "<redacted>"")\215P\256\002\302\361\025)\236\rI]\255\241^\250\030+\004\3636!j\276lc$\025\300l\027N\275O\035T\274\302\3555\200\216\322\212\242\210\302[\342r\211s\325PR\200\310M\227\201x\204\354" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "f38c4acc68bb473786d01674d0ae9341"
I20260804 14:52:01.514304 28473 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:01.547668 28494 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=2, attempt_no=0}) took 1972 ms. Trace:
I20260804 14:52:01.547840 28494 rpcz_store.cc:276] 0804 14:51:59.575070 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:51:59.575140 (+ 70us) service_pool.cc:227] Handling call
0804 14:52:01.547647 (+1972507us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:00.171360 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:00.171484 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:00.171489 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:52:00.171490 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:00.171578 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:00.171588 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:00.171590 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:00.171637 (+ 47us) tablet.cc:699] Row locks acquired
0804 14:52:00.171639 (+ 2us) write_op.cc:260] PREPARE: finished
0804 14:52:00.171685 (+ 46us) write_op.cc:270] Start()
0804 14:52:00.924833 (+753148us) write_op.cc:276] Timestamp: P: 1785855120171609 usec, L: 0
0804 14:52:00.924837 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:52:00.925078 (+ 241us) log.cc:842] Serialized 354 byte log entry
0804 14:52:01.546833 (+621755us) op_driver.cc:444] REPLICATION: finished
0804 14:52:01.547157 (+ 324us) write_op.cc:301] APPLY: starting
0804 14:52:01.547181 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:01.547240 (+ 59us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:01.547255 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:01.547324 (+ 69us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:01.547359 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:01.547364 (+ 5us) write_op.cc:312] APPLY: finished
0804 14:52:01.547458 (+ 94us) log.cc:842] Serialized 22 byte log entry
0804 14:52:01.547541 (+ 83us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:01.547558 (+ 17us) write_op.cc:454] Released schema lock
0804 14:52:01.547564 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":252,"num_ops":1,"prepare.queue_time_us":595824,"prepare.run_cpu_time_us":730,"prepare.run_wall_time_us":753804,"replication_time_us":621960,"thread_start_us":104,"threads_started":1}]]}
I20260804 14:52:01.826838 28486 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:01.826999 28486 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:01.827643 28486 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:52:01.828024 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:01.828095 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:52:01.828809 27893 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:01.981611 28473 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:01.981786 28473 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:01.982487 28473 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:01.982735 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:01.982816 27988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:01.983438 28253 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:01.983713 28473 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:02.015929 28494 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=3, attempt_no=0}) took 1552 ms. Trace:
I20260804 14:52:02.016134 28494 rpcz_store.cc:276] 0804 14:52:00.463881 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:00.463925 (+ 44us) service_pool.cc:227] Handling call
0804 14:52:02.015913 (+1551988us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:00.925196 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:00.925308 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:00.925313 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:52:00.925315 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:00.925400 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:00.925410 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:00.925412 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:00.925472 (+ 60us) tablet.cc:699] Row locks acquired
0804 14:52:00.925473 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:00.925536 (+ 63us) write_op.cc:270] Start()
0804 14:52:01.660755 (+735219us) write_op.cc:276] Timestamp: P: 1785855120925500 usec, L: 0
0804 14:52:01.660759 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:52:01.661045 (+ 286us) log.cc:842] Serialized 354 byte log entry
0804 14:52:02.015331 (+354286us) op_driver.cc:444] REPLICATION: finished
0804 14:52:02.015493 (+ 162us) write_op.cc:301] APPLY: starting
0804 14:52:02.015518 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:02.015558 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:02.015580 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:02.015649 (+ 69us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:02.015689 (+ 40us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:02.015693 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:52:02.015768 (+ 75us) log.cc:842] Serialized 22 byte log entry
0804 14:52:02.015829 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:02.015842 (+ 13us) write_op.cc:454] Released schema lock
0804 14:52:02.015847 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":54,"num_ops":1,"prepare.queue_time_us":460772,"prepare.run_cpu_time_us":897,"prepare.run_wall_time_us":735952,"raft.queue_time_us":45,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":166,"replication_time_us":354533,"spinlock_wait_cycles":56320}]]}
W20260804 14:52:02.217502 28283 tablet_service.cc:2970] error setting up scanner 27091456997643048168a92eb1ecf16a with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.234s (maximum allowed 0.100s): new_scan_request { tablet_id: "6cd92ff096234dafa214acc72d617f73" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\273\373\307\323\006\"3\n\005slave\022*\n 41255276b38541d68bbfe0d0057a0118\020\001\030\001 \001(\001" signature: "<redacted>"")\215P\256\002\302\361\025)\236\rI]\255\241^\250\030+\004\3636!j\276lc$\025\300l\027N\275O\035T\274\302\3555\200\216\322\212\242\210\302[\342r\211s\325PR\200\310M\227\201x\204\354" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "a778d080873b4800b57ddf4a1fab5db6"
I20260804 14:52:02.626641 28486 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:02.648257 28486 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:02.648420 28486 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:02.649020 28486 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:52:02.649425 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:02.649674 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:52:02.650082 27893 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:02.650272 28486 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:02.665539 28504 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=4, attempt_no=0}) took 1116 ms. Trace:
I20260804 14:52:02.665673 28504 rpcz_store.cc:276] 0804 14:52:01.549545 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:01.549612 (+ 67us) service_pool.cc:227] Handling call
0804 14:52:02.665517 (+1115905us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:01.661185 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:01.661297 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:01.661303 (+ 6us) write_op.cc:435] Acquired schema lock
0804 14:52:01.661305 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:01.661399 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:01.661413 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:01.661419 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:01.661469 (+ 50us) tablet.cc:699] Row locks acquired
0804 14:52:01.661470 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:01.661519 (+ 49us) write_op.cc:270] Start()
0804 14:52:02.594687 (+933168us) write_op.cc:276] Timestamp: P: 1785855121661462 usec, L: 0
0804 14:52:02.594693 (+ 6us) op_driver.cc:331] REPLICATION: starting
0804 14:52:02.594898 (+ 205us) log.cc:842] Serialized 354 byte log entry
0804 14:52:02.664597 (+ 69699us) op_driver.cc:444] REPLICATION: finished
0804 14:52:02.664959 (+ 362us) write_op.cc:301] APPLY: starting
0804 14:52:02.665032 (+ 73us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:02.665077 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:02.665107 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:02.665198 (+ 91us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:02.665228 (+ 30us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:02.665231 (+ 3us) write_op.cc:312] APPLY: finished
0804 14:52:02.665316 (+ 85us) log.cc:842] Serialized 22 byte log entry
0804 14:52:02.665412 (+ 96us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:02.665426 (+ 14us) write_op.cc:454] Released schema lock
0804 14:52:02.665431 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":291,"num_ops":1,"prepare.queue_time_us":111228,"prepare.run_cpu_time_us":857,"prepare.run_wall_time_us":933932,"raft.queue_time_us":116,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":4,"replication_time_us":69857,"spinlock_wait_cycles":11648,"thread_start_us":230,"threads_started":2}]]}
I20260804 14:52:03.066848 28486 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:03.067021 28486 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:03.067680 28486 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:52:03.068063 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:03.068248 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:52:03.068687 27893 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:03.068917 28486 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:03.110497 28504 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=5, attempt_no=0}) took 1092 ms. Trace:
I20260804 14:52:03.110642 28504 rpcz_store.cc:276] 0804 14:52:02.017947 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:02.018000 (+ 53us) service_pool.cc:227] Handling call
0804 14:52:03.110479 (+1092479us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:02.595152 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:02.595300 (+ 148us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:02.595305 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:52:02.595307 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:02.595389 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:02.595404 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:02.595406 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:02.595444 (+ 38us) tablet.cc:699] Row locks acquired
0804 14:52:02.595445 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:02.595484 (+ 39us) write_op.cc:270] Start()
0804 14:52:02.654702 (+ 59218us) write_op.cc:276] Timestamp: P: 1785855122595438 usec, L: 0
0804 14:52:02.654705 (+ 3us) op_driver.cc:331] REPLICATION: starting
0804 14:52:02.654905 (+ 200us) log.cc:842] Serialized 354 byte log entry
0804 14:52:03.109939 (+455034us) op_driver.cc:444] REPLICATION: finished
0804 14:52:03.110032 (+ 93us) write_op.cc:301] APPLY: starting
0804 14:52:03.110083 (+ 51us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:03.110113 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:03.110127 (+ 14us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:03.110182 (+ 55us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:03.110209 (+ 27us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:03.110212 (+ 3us) write_op.cc:312] APPLY: finished
0804 14:52:03.110284 (+ 72us) log.cc:842] Serialized 22 byte log entry
0804 14:52:03.110377 (+ 93us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:03.110392 (+ 15us) write_op.cc:454] Released schema lock
0804 14:52:03.110398 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":33,"num_ops":1,"prepare.queue_time_us":576637,"prepare.run_cpu_time_us":732,"prepare.run_wall_time_us":59826,"replication_time_us":455203}]]}
I20260804 14:52:03.126642 28500 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:03.126804 28500 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:03.127470 28500 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:03.127727 27989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:03.128242 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:03.128727 28252 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:03.714702 28500 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260804 14:52:03.903390 28283 tablet_service.cc:2970] error setting up scanner e04a404d1dbf4da8a14d7d842ea6ea3a with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.189s (maximum allowed 0.100s): new_scan_request { tablet_id: "6cd92ff096234dafa214acc72d617f73" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\273\373\307\323\006\"3\n\005slave\022*\n 41255276b38541d68bbfe0d0057a0118\020\001\030\001 \001(\001" signature: "<redacted>"")\215P\256\002\302\361\025)\236\rI]\255\241^\250\030+\004\3636!j\276lc$\025\300l\027N\275O\035T\274\302\3555\200\216\322\212\242\210\302[\342r\211s\325PR\200\310M\227\201x\204\354" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "f38c4acc68bb473786d01674d0ae9341"
I20260804 14:52:03.999078 28500 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:03.999289 28500 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:03.999893 28500 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:04.000389 27988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:04.000528 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:04.001179 28252 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:04.001422 28500 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:04.022054 28517 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=6, attempt_no=0}) took 1354 ms. Trace:
I20260804 14:52:04.022158 28517 rpcz_store.cc:276] 0804 14:52:02.667518 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:02.667595 (+ 77us) service_pool.cc:227] Handling call
0804 14:52:04.022037 (+1354442us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:02.668121 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:02.668223 (+ 102us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:02.668226 (+ 3us) write_op.cc:435] Acquired schema lock
0804 14:52:02.668228 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:02.668295 (+ 67us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:02.668303 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:02.668305 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:02.668340 (+ 35us) tablet.cc:699] Row locks acquired
0804 14:52:02.668341 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:02.668391 (+ 50us) write_op.cc:270] Start()
0804 14:52:03.268546 (+600155us) write_op.cc:276] Timestamp: P: 1785855122668322 usec, L: 0
0804 14:52:03.268550 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:52:03.268730 (+ 180us) log.cc:842] Serialized 354 byte log entry
0804 14:52:04.021375 (+752645us) op_driver.cc:444] REPLICATION: finished
0804 14:52:04.021669 (+ 294us) write_op.cc:301] APPLY: starting
0804 14:52:04.021689 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:04.021728 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:04.021742 (+ 14us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:04.021808 (+ 66us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:04.021835 (+ 27us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:04.021837 (+ 2us) write_op.cc:312] APPLY: finished
0804 14:52:04.021904 (+ 67us) log.cc:842] Serialized 22 byte log entry
0804 14:52:04.021958 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:04.021970 (+ 12us) write_op.cc:454] Released schema lock
0804 14:52:04.021974 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":226,"num_ops":1,"prepare.queue_time_us":61,"prepare.run_cpu_time_us":614,"prepare.run_wall_time_us":600694,"replication_time_us":752788,"thread_start_us":108,"threads_started":1}]]}
I20260804 14:52:04.216602 28486 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:04.216774 28486 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:04.217382 28486 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:52:04.217826 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:52:04.217834 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:04.218461 27893 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:04.683389 28500 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:04.683553 28500 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:04.684316 28500 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:04.684748 27988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:04.684888 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:04.685323 28252 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:04.718600 28525 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=7, attempt_no=0}) took 1606 ms. Trace:
I20260804 14:52:04.718758 28525 rpcz_store.cc:276] 0804 14:52:03.112196 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:03.112254 (+ 58us) service_pool.cc:227] Handling call
0804 14:52:04.718575 (+1606321us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:03.268859 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:03.268971 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:03.268978 (+ 7us) write_op.cc:435] Acquired schema lock
0804 14:52:03.268980 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:03.269059 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:03.269068 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:03.269071 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:03.269119 (+ 48us) tablet.cc:699] Row locks acquired
0804 14:52:03.269120 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:03.269201 (+ 81us) write_op.cc:270] Start()
0804 14:52:04.126378 (+857177us) write_op.cc:276] Timestamp: P: 1785855123269143 usec, L: 0
0804 14:52:04.126383 (+ 5us) op_driver.cc:331] REPLICATION: starting
0804 14:52:04.126583 (+ 200us) log.cc:842] Serialized 354 byte log entry
0804 14:52:04.717708 (+591125us) op_driver.cc:444] REPLICATION: finished
0804 14:52:04.718063 (+ 355us) write_op.cc:301] APPLY: starting
0804 14:52:04.718085 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:04.718120 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:04.718167 (+ 47us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:04.718234 (+ 67us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:04.718262 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:04.718265 (+ 3us) write_op.cc:312] APPLY: finished
0804 14:52:04.718343 (+ 78us) log.cc:842] Serialized 22 byte log entry
0804 14:52:04.718451 (+ 108us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:04.718467 (+ 16us) write_op.cc:454] Released schema lock
0804 14:52:04.718472 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":286,"num_ops":1,"prepare.queue_time_us":156150,"prepare.run_cpu_time_us":772,"prepare.run_wall_time_us":857829,"raft.queue_time_us":69,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":149,"replication_time_us":591282,"thread_start_us":137,"threads_started":1}]]}
I20260804 14:52:04.842449 28500 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:04.880827 28486 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:05.467358 28500 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:05.467512 28500 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:05.468061 28500 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:05.468580 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:05.468580 27988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:05.469046 28253 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
W20260804 14:52:06.059717 28283 tablet_service.cc:2970] error setting up scanner 4833928b553043bfb9a598e799bfecc8 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.590s (maximum allowed 0.100s): new_scan_request { tablet_id: "6cd92ff096234dafa214acc72d617f73" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\273\373\307\323\006\"3\n\005slave\022*\n 41255276b38541d68bbfe0d0057a0118\020\001\030\001 \001(\001" signature: "<redacted>"")\215P\256\002\302\361\025)\236\rI]\255\241^\250\030+\004\3636!j\276lc$\025\300l\027N\275O\035T\274\302\3555\200\216\322\212\242\210\302[\342r\211s\325PR\200\310M\227\201x\204\354" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "a778d080873b4800b57ddf4a1fab5db6"
I20260804 14:52:06.099795 28486 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:06.099960 28486 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:06.100157 27989 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.24.6.130:53965 (request call id 1365) took 1217 ms. Trace:
I20260804 14:52:06.100507 28486 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:52:06.101061 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:06.101081 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:52:06.101543 27893 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:06.100267 27989 rpcz_store.cc:276] 0804 14:52:04.882217 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:04.882270 (+ 53us) service_pool.cc:227] Handling call
0804 14:52:04.882429 (+ 159us) raft_consensus.cc:1457] Updating replica for 2 ops
0804 14:52:04.882511 (+ 82us) raft_consensus.cc:1520] Early marking committed up to index 9
0804 14:52:04.882652 (+ 141us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0804 14:52:04.882751 (+ 99us) write_op.cc:270] Start()
0804 14:52:05.658937 (+776186us) write_op.cc:276] Timestamp: P: 1785855124126968 usec, L: 0
0804 14:52:05.659520 (+ 583us) write_op.cc:270] Start()
0804 14:52:06.098793 (+439273us) write_op.cc:276] Timestamp: P: 1785855124720832 usec, L: 0
0804 14:52:06.099302 (+ 509us) log.cc:842] Serialized 708 byte log entry
0804 14:52:06.099522 (+ 220us) raft_consensus.cc:1631] Marking committed up to 10
0804 14:52:06.099699 (+ 177us) raft_consensus.cc:1683] Filling consensus response to leader.
0804 14:52:06.099737 (+ 38us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0804 14:52:06.100073 (+ 336us) raft_consensus.cc:1671] finished
0804 14:52:06.100077 (+ 4us) raft_consensus.cc:1677] UpdateReplicas() finished
0804 14:52:06.100135 (+ 58us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:05.659445 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:05.659566 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:05.659570 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:52:05.659571 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:05.659644 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:05.659658 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:05.659661 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:05.659702 (+ 41us) tablet.cc:699] Row locks acquired
0804 14:52:05.659703 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:06.099562 (+439859us) op_driver.cc:444] REPLICATION: finished
0804 14:52:06.099810 (+ 248us) write_op.cc:301] APPLY: starting
0804 14:52:06.099830 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:06.099867 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:06.099869 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:06.099933 (+ 64us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:06.099963 (+ 30us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:06.099990 (+ 27us) write_op.cc:312] APPLY: finished
0804 14:52:06.100053 (+ 63us) log.cc:842] Serialized 22 byte log entry
0804 14:52:06.100141 (+ 88us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:06.100153 (+ 12us) write_op.cc:454] Released schema lock
0804 14:52:06.100159 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0804 14:52:06.100234 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:06.100236 (+ 2us) write_op.cc:454] Released schema lock
Related trace 'op':
0804 14:52:06.099119 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:06.099310 (+ 191us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:06.099315 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:52:06.099317 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:06.099397 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:06.099434 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:06.099436 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:06.099483 (+ 47us) tablet.cc:699] Row locks acquired
0804 14:52:06.099484 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":140,"threads_started":1,"wal-append.queue_time_us":228,"child_traces":[["op",{"apply.queue_time_us":189,"apply.run_cpu_time_us":470,"apply.run_wall_time_us":470,"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":323,"prepare.run_cpu_time_us":374,"prepare.run_wall_time_us":374,"replication_time_us":1216793,"thread_start_us":271,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":31,"prepare.queue_time_us":106,"prepare.run_cpu_time_us":467,"prepare.run_wall_time_us":468}]]}
I20260804 14:52:06.101816 28486 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:06.204615 28500 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:06.247902 28548 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=10, attempt_no=0}) took 1362 ms. Trace:
I20260804 14:52:06.248036 28548 rpcz_store.cc:276] 0804 14:52:04.885225 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:04.885346 (+ 121us) service_pool.cc:227] Handling call
0804 14:52:06.247883 (+1362537us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:04.885810 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:04.885910 (+ 100us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:04.885913 (+ 3us) write_op.cc:435] Acquired schema lock
0804 14:52:04.885914 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:04.885979 (+ 65us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:04.885985 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:04.885987 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:04.886021 (+ 34us) tablet.cc:699] Row locks acquired
0804 14:52:04.886022 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:04.886053 (+ 31us) write_op.cc:270] Start()
0804 14:52:05.016216 (+130163us) write_op.cc:276] Timestamp: P: 1785855124886012 usec, L: 0
0804 14:52:05.016220 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:52:05.016454 (+ 234us) log.cc:842] Serialized 354 byte log entry
0804 14:52:06.247053 (+1230599us) op_driver.cc:444] REPLICATION: finished
0804 14:52:06.247411 (+ 358us) write_op.cc:301] APPLY: starting
0804 14:52:06.247438 (+ 27us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:06.247479 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:06.247502 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:06.247594 (+ 92us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:06.247634 (+ 40us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:06.247638 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:52:06.247722 (+ 84us) log.cc:842] Serialized 22 byte log entry
0804 14:52:06.247784 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:06.247807 (+ 23us) write_op.cc:454] Released schema lock
0804 14:52:06.247812 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":275,"num_ops":1,"prepare.queue_time_us":42,"prepare.run_cpu_time_us":646,"prepare.run_wall_time_us":130713,"replication_time_us":1230799,"thread_start_us":159,"threads_started":1}]]}
W20260804 14:52:06.721765 28283 tablet_service.cc:2970] error setting up scanner 3979b38b05f346028d8bf1e1db3c19b2 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.515s (maximum allowed 0.100s): new_scan_request { tablet_id: "6cd92ff096234dafa214acc72d617f73" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\273\373\307\323\006\"3\n\005slave\022*\n 41255276b38541d68bbfe0d0057a0118\020\001\030\001 \001(\001" signature: "<redacted>"")\215P\256\002\302\361\025)\236\rI]\255\241^\250\030+\004\3636!j\276lc$\025\300l\027N\275O\035T\274\302\3555\200\216\322\212\242\210\302[\342r\211s\325PR\200\310M\227\201x\204\354" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "f38c4acc68bb473786d01674d0ae9341"
I20260804 14:52:06.727160 28486 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:06.727368 28486 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:06.727967 28486 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:52:06.728317 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:06.728490 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:52:06.729024 27891 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:06.798103 28500 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:06.798244 28500 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:06.798781 28500 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:06.799722 27988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:06.800486 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:06.800920 28252 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:06.842952 28555 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=11, attempt_no=0}) took 1347 ms. Trace:
I20260804 14:52:06.843146 28555 rpcz_store.cc:276] 0804 14:52:05.495196 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:05.495324 (+ 128us) service_pool.cc:227] Handling call
0804 14:52:06.842930 (+1347606us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:05.495939 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:05.496030 (+ 91us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:05.496033 (+ 3us) write_op.cc:435] Acquired schema lock
0804 14:52:05.496034 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:05.496099 (+ 65us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:05.496107 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:05.496109 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:05.496144 (+ 35us) tablet.cc:699] Row locks acquired
0804 14:52:05.496145 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:05.496182 (+ 37us) write_op.cc:270] Start()
0804 14:52:06.458312 (+962130us) write_op.cc:276] Timestamp: P: 1785855125496131 usec, L: 0
0804 14:52:06.458315 (+ 3us) op_driver.cc:331] REPLICATION: starting
0804 14:52:06.458486 (+ 171us) log.cc:842] Serialized 354 byte log entry
0804 14:52:06.842068 (+383582us) op_driver.cc:444] REPLICATION: finished
0804 14:52:06.842420 (+ 352us) write_op.cc:301] APPLY: starting
0804 14:52:06.842445 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:06.842479 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:06.842499 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:06.842568 (+ 69us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:06.842608 (+ 40us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:06.842611 (+ 3us) write_op.cc:312] APPLY: finished
0804 14:52:06.842691 (+ 80us) log.cc:842] Serialized 22 byte log entry
0804 14:52:06.842785 (+ 94us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:06.842797 (+ 12us) write_op.cc:454] Released schema lock
0804 14:52:06.842803 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":271,"num_ops":1,"prepare.queue_time_us":155,"prepare.run_cpu_time_us":546,"prepare.run_wall_time_us":962605,"replication_time_us":383698,"thread_start_us":118,"threads_started":1}]]}
I20260804 14:52:07.249795 28500 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:07.276618 28500 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:07.276791 28500 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:07.277333 28500 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:07.277802 27988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:07.277930 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:07.278544 28252 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:07.278751 28500 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:07.294544 28555 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=12, attempt_no=0}) took 1044 ms. Trace:
I20260804 14:52:07.294665 28555 rpcz_store.cc:276] 0804 14:52:06.249668 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:06.249754 (+ 86us) service_pool.cc:227] Handling call
0804 14:52:07.294524 (+1044770us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:06.458574 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:06.458666 (+ 92us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:06.458670 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:52:06.458671 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:06.458736 (+ 65us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:06.458744 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:06.458746 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:06.458783 (+ 37us) tablet.cc:699] Row locks acquired
0804 14:52:06.458784 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:06.458818 (+ 34us) write_op.cc:270] Start()
0804 14:52:06.703055 (+244237us) write_op.cc:276] Timestamp: P: 1785855126458791 usec, L: 0
0804 14:52:06.703059 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:52:06.703326 (+ 267us) log.cc:842] Serialized 354 byte log entry
0804 14:52:07.293905 (+590579us) op_driver.cc:444] REPLICATION: finished
0804 14:52:07.294057 (+ 152us) write_op.cc:301] APPLY: starting
0804 14:52:07.294081 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:07.294112 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:07.294125 (+ 13us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:07.294185 (+ 60us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:07.294212 (+ 27us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:07.294215 (+ 3us) write_op.cc:312] APPLY: finished
0804 14:52:07.294314 (+ 99us) log.cc:842] Serialized 22 byte log entry
0804 14:52:07.294413 (+ 99us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:07.294426 (+ 13us) write_op.cc:454] Released schema lock
0804 14:52:07.294432 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":85,"num_ops":1,"prepare.queue_time_us":208365,"prepare.run_cpu_time_us":713,"prepare.run_wall_time_us":244809,"replication_time_us":590808}]]}
I20260804 14:52:07.458712 28486 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:08.007546 28486 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:08.007690 28486 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:08.008208 28486 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:52:08.008663 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:52:08.008663 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:08.009232 27891 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:08.009524 28486 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:08.164069 28486 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:08.164242 28486 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:08.164727 28486 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:52:08.165163 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:08.165159 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:52:08.165752 27893 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:08.165961 28486 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:08.638947 28562 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:08.639117 28562 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:08.639978 28562 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:08.640455 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:08.640422 27988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:08.640967 28253 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:08.651480 28567 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=14, attempt_no=0}) took 1355 ms. Trace:
I20260804 14:52:08.651605 28567 rpcz_store.cc:276] 0804 14:52:07.296246 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:07.296300 (+ 54us) service_pool.cc:227] Handling call
0804 14:52:08.651444 (+1355144us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:07.296701 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:07.296811 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:07.296815 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:52:07.296816 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:07.296891 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:07.296900 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:07.296902 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:07.296948 (+ 46us) tablet.cc:699] Row locks acquired
0804 14:52:07.296949 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:07.296975 (+ 26us) write_op.cc:270] Start()
0804 14:52:08.225119 (+928144us) write_op.cc:276] Timestamp: P: 1785855127296903 usec, L: 0
0804 14:52:08.225123 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:52:08.225326 (+ 203us) log.cc:842] Serialized 354 byte log entry
0804 14:52:08.650443 (+425117us) op_driver.cc:444] REPLICATION: finished
0804 14:52:08.650762 (+ 319us) write_op.cc:301] APPLY: starting
0804 14:52:08.650782 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:08.650816 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:08.650834 (+ 18us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:08.651058 (+ 224us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:08.651087 (+ 29us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:08.651090 (+ 3us) write_op.cc:312] APPLY: finished
0804 14:52:08.651179 (+ 89us) log.cc:842] Serialized 22 byte log entry
0804 14:52:08.651316 (+ 137us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:08.651332 (+ 16us) write_op.cc:454] Released schema lock
0804 14:52:08.651338 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":248,"num_ops":1,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":630,"prepare.run_wall_time_us":928693,"replication_time_us":425276,"thread_start_us":108,"threads_started":1}]]}
I20260804 14:52:08.875929 28486 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:08.876101 28486 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:08.876693 28486 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:52:08.877089 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:08.877192 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:52:08.877662 27893 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:09.055275 28562 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:09.073423 28567 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=15, attempt_no=0}) took 1734 ms. Trace:
I20260804 14:52:09.073585 28567 rpcz_store.cc:276] 0804 14:52:07.339077 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:07.339253 (+ 176us) service_pool.cc:227] Handling call
0804 14:52:09.073402 (+1734149us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:08.225427 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:08.225555 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:08.225560 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:52:08.225562 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:08.225646 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:08.225655 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:08.225658 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:08.225703 (+ 45us) tablet.cc:699] Row locks acquired
0804 14:52:08.225704 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:08.225751 (+ 47us) write_op.cc:270] Start()
0804 14:52:08.527944 (+302193us) write_op.cc:276] Timestamp: P: 1785855128225690 usec, L: 0
0804 14:52:08.527950 (+ 6us) op_driver.cc:331] REPLICATION: starting
0804 14:52:08.528217 (+ 267us) log.cc:842] Serialized 354 byte log entry
0804 14:52:09.072714 (+544497us) op_driver.cc:444] REPLICATION: finished
0804 14:52:09.072883 (+ 169us) write_op.cc:301] APPLY: starting
0804 14:52:09.072911 (+ 28us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:09.072945 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:09.072967 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:09.073073 (+ 106us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:09.073106 (+ 33us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:09.073111 (+ 5us) write_op.cc:312] APPLY: finished
0804 14:52:09.073194 (+ 83us) log.cc:842] Serialized 22 byte log entry
0804 14:52:09.073282 (+ 88us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:09.073297 (+ 15us) write_op.cc:454] Released schema lock
0804 14:52:09.073302 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":73,"num_ops":1,"prepare.queue_time_us":885784,"prepare.run_cpu_time_us":807,"prepare.run_wall_time_us":302889,"replication_time_us":544715}]]}
I20260804 14:52:09.075462 28486 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:09.506129 28562 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:09.506317 28562 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:09.506982 28562 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:09.507443 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:09.507529 27988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:09.508203 28252 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:09.508560 28562 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:09.826026 28486 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:09.826167 28486 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:09.826680 28486 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:52:09.827114 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:52:09.827188 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:09.827754 27891 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:09.827939 28486 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:09.944120 28562 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:09.944286 28562 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:09.944877 28562 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:09.945286 27988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:09.945284 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:09.945725 28252 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:09.945910 28562 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:10.073205 28486 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:10.073357 28486 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:10.073936 28486 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:52:10.074285 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:10.074301 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:52:10.074710 27891 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:10.074935 28486 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:10.163106 28562 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:10.163300 28562 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:10.163770 28562 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:10.164098 27988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:10.164160 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:10.164690 28253 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:10.164896 28562 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260804 14:52:10.539788 28283 tablet_service.cc:2970] error setting up scanner 97f0e562bacf4b308924102d6876514b with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1785855129539468 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1785855129076000 usec, L: 0 Physical time difference: 0.463s: new_scan_request { tablet_id: "6cd92ff096234dafa214acc72d617f73" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\273\373\307\323\006\"3\n\005slave\022*\n 41255276b38541d68bbfe0d0057a0118\020\001\030\001 \001(\001" signature: "<redacted>"")\215P\256\002\302\361\025)\236\rI]\255\241^\250\030+\004\3636!j\276lc$\025\300l\027N\275O\035T\274\302\3555\200\216\322\212\242\210\302[\342r\211s\325PR\200\310M\227\201x\204\354" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "f38c4acc68bb473786d01674d0ae9341"
I20260804 14:52:10.540148 28283 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:43512 (request call id 18) took 1001 ms. Trace:
I20260804 14:52:10.540257 28283 rpcz_store.cc:276] 0804 14:52:09.539087 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:09.539132 (+ 45us) service_pool.cc:227] Handling call
0804 14:52:09.539308 (+ 176us) tablet_service.cc:2780] Created scanner 97f0e562bacf4b308924102d6876514b for tablet 6cd92ff096234dafa214acc72d617f73, query id is f38c4acc68bb473786d01674d0ae9341
0804 14:52:09.539514 (+ 206us) tablet_service.cc:2920] Creating iterator
0804 14:52:09.539535 (+ 21us) tablet_service.cc:3297] Waiting safe time to advance
0804 14:52:10.539761 (+1000226us) tablet_service.cc:2945] Iterator created
0804 14:52:10.539776 (+ 15us) tablet_service.cc:2967] Iterator init: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1785855129539468 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1785855129076000 usec, L: 0 Physical time difference: 0.463s
0804 14:52:10.540137 (+ 361us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260804 14:52:10.670006 28283 tablet_service.cc:2970] error setting up scanner 94cc36372aec4e76aa9d02fa463d1f3f with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.724s (maximum allowed 0.100s): new_scan_request { tablet_id: "6cd92ff096234dafa214acc72d617f73" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\273\373\307\323\006\"3\n\005slave\022*\n 41255276b38541d68bbfe0d0057a0118\020\001\030\001 \001(\001" signature: "<redacted>"")\215P\256\002\302\361\025)\236\rI]\255\241^\250\030+\004\3636!j\276lc$\025\300l\027N\275O\035T\274\302\3555\200\216\322\212\242\210\302[\342r\211s\325PR\200\310M\227\201x\204\354" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "a778d080873b4800b57ddf4a1fab5db6"
I20260804 14:52:11.185408 28486 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:11.185573 28486 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:11.186091 28486 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:52:11.186671 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:11.186674 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:52:11.187222 27893 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:11.187409 28486 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:11.213577 28585 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=18, attempt_no=0}) took 1675 ms. Trace:
I20260804 14:52:11.213689 28585 rpcz_store.cc:276] 0804 14:52:09.538149 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:09.538238 (+ 89us) service_pool.cc:227] Handling call
0804 14:52:11.213537 (+1675299us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:09.641603 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:09.641720 (+ 117us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:09.641725 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:52:09.641726 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:09.641826 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:09.641840 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:09.641846 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:09.641906 (+ 60us) tablet.cc:699] Row locks acquired
0804 14:52:09.641909 (+ 3us) write_op.cc:260] PREPARE: finished
0804 14:52:09.641962 (+ 53us) write_op.cc:270] Start()
0804 14:52:10.473146 (+831184us) write_op.cc:276] Timestamp: P: 1785855129641914 usec, L: 0
0804 14:52:10.473150 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:52:10.473365 (+ 215us) log.cc:842] Serialized 354 byte log entry
0804 14:52:11.212539 (+739174us) op_driver.cc:444] REPLICATION: finished
0804 14:52:11.212883 (+ 344us) write_op.cc:301] APPLY: starting
0804 14:52:11.212947 (+ 64us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:11.212985 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:11.213000 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:11.213063 (+ 63us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:11.213094 (+ 31us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:11.213097 (+ 3us) write_op.cc:312] APPLY: finished
0804 14:52:11.213357 (+ 260us) log.cc:842] Serialized 22 byte log entry
0804 14:52:11.213427 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:11.213440 (+ 13us) write_op.cc:454] Released schema lock
0804 14:52:11.213446 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":276,"num_ops":1,"prepare.queue_time_us":102922,"prepare.run_cpu_time_us":778,"prepare.run_wall_time_us":831836,"replication_time_us":739355,"thread_start_us":184,"threads_started":1}]]}
I20260804 14:52:11.382427 28562 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:11.382587 28562 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:11.383289 28562 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:11.383869 27989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:11.383945 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:11.384351 28252 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:11.384574 28562 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:11.693686 28486 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:11.693845 28486 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:11.694399 28486 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:52:11.694815 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:11.695026 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:52:11.695456 27893 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:11.695654 28486 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:11.725555 28592 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=19, attempt_no=0}) took 1741 ms. Trace:
I20260804 14:52:11.725700 28592 rpcz_store.cc:276] 0804 14:52:09.984332 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:09.984402 (+ 70us) service_pool.cc:227] Handling call
0804 14:52:11.725536 (+1741134us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:10.473483 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:10.473606 (+ 123us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:10.473612 (+ 6us) write_op.cc:435] Acquired schema lock
0804 14:52:10.473613 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:10.473702 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:10.473714 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:10.473717 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:10.473768 (+ 51us) tablet.cc:699] Row locks acquired
0804 14:52:10.473770 (+ 2us) write_op.cc:260] PREPARE: finished
0804 14:52:10.473820 (+ 50us) write_op.cc:270] Start()
0804 14:52:11.304024 (+830204us) write_op.cc:276] Timestamp: P: 1785855130473795 usec, L: 0
0804 14:52:11.304030 (+ 6us) op_driver.cc:331] REPLICATION: starting
0804 14:52:11.304257 (+ 227us) log.cc:842] Serialized 354 byte log entry
0804 14:52:11.724734 (+420477us) op_driver.cc:444] REPLICATION: finished
0804 14:52:11.725056 (+ 322us) write_op.cc:301] APPLY: starting
0804 14:52:11.725081 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:11.725121 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:11.725139 (+ 18us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:11.725242 (+ 103us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:11.725290 (+ 48us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:11.725294 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:52:11.725386 (+ 92us) log.cc:842] Serialized 22 byte log entry
0804 14:52:11.725448 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:11.725461 (+ 13us) write_op.cc:454] Released schema lock
0804 14:52:11.725465 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":254,"num_ops":1,"prepare.queue_time_us":488686,"prepare.run_cpu_time_us":783,"prepare.run_wall_time_us":830844,"replication_time_us":420674,"thread_start_us":143,"threads_started":1}]]}
I20260804 14:52:11.797271 28562 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:11.797530 28562 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:11.798247 28562 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:11.798964 27988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:11.799045 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:11.801008 28252 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:11.801246 28562 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:12.300603 28595 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:12.300930 28595 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:12.301538 28595 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:52:12.301978 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:12.301980 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:52:12.302440 27893 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:12.302654 28595 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:12.318744 28600 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=20, attempt_no=0}) took 1103 ms. Trace:
I20260804 14:52:12.318846 28600 rpcz_store.cc:276] 0804 14:52:11.215403 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:11.215473 (+ 70us) service_pool.cc:227] Handling call
0804 14:52:12.318724 (+1103251us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:11.304358 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:11.304471 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:11.304478 (+ 7us) write_op.cc:435] Acquired schema lock
0804 14:52:11.304480 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:11.304574 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:11.304585 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:11.304588 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:11.304634 (+ 46us) tablet.cc:699] Row locks acquired
0804 14:52:11.304635 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:11.304684 (+ 49us) write_op.cc:270] Start()
0804 14:52:12.244816 (+940132us) write_op.cc:276] Timestamp: P: 1785855131304620 usec, L: 0
0804 14:52:12.244820 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:52:12.245013 (+ 193us) log.cc:842] Serialized 354 byte log entry
0804 14:52:12.317921 (+ 72908us) op_driver.cc:444] REPLICATION: finished
0804 14:52:12.318218 (+ 297us) write_op.cc:301] APPLY: starting
0804 14:52:12.318245 (+ 27us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:12.318277 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:12.318290 (+ 13us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:12.318405 (+ 115us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:12.318441 (+ 36us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:12.318447 (+ 6us) write_op.cc:312] APPLY: finished
0804 14:52:12.318544 (+ 97us) log.cc:842] Serialized 22 byte log entry
0804 14:52:12.318617 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:12.318636 (+ 19us) write_op.cc:454] Released schema lock
0804 14:52:12.318642 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":240,"num_ops":1,"prepare.queue_time_us":88385,"prepare.run_cpu_time_us":690,"prepare.run_wall_time_us":940745,"raft.queue_time_us":24,"raft.run_cpu_time_us":65,"raft.run_wall_time_us":66,"replication_time_us":73064,"spinlock_wait_cycles":139008,"thread_start_us":116,"threads_started":1}]]}
I20260804 14:52:12.425220 28562 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:12.425388 28562 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:12.425933 28562 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:12.426777 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:12.426777 27989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:12.427354 28252 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:12.427608 28562 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:13.066705 28595 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:13.066874 28595 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:13.067543 28595 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:52:13.067845 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:13.068061 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:52:13.068485 27893 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:13.086901 28610 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=21, attempt_no=0}) took 1359 ms. Trace:
I20260804 14:52:13.087062 28610 rpcz_store.cc:276] 0804 14:52:11.727088 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:11.727167 (+ 79us) service_pool.cc:227] Handling call
0804 14:52:13.086876 (+1359709us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:12.245143 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:12.245260 (+ 117us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:12.245265 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:52:12.245267 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:12.245369 (+ 102us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:12.245379 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:12.245382 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:12.245425 (+ 43us) tablet.cc:699] Row locks acquired
0804 14:52:12.245434 (+ 9us) write_op.cc:260] PREPARE: finished
0804 14:52:12.245480 (+ 46us) write_op.cc:270] Start()
0804 14:52:12.530632 (+285152us) write_op.cc:276] Timestamp: P: 1785855132245432 usec, L: 0
0804 14:52:12.530637 (+ 5us) op_driver.cc:331] REPLICATION: starting
0804 14:52:12.530820 (+ 183us) log.cc:842] Serialized 354 byte log entry
0804 14:52:13.086051 (+555231us) op_driver.cc:444] REPLICATION: finished
0804 14:52:13.086385 (+ 334us) write_op.cc:301] APPLY: starting
0804 14:52:13.086409 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:13.086445 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:13.086467 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:13.086537 (+ 70us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:13.086566 (+ 29us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:13.086569 (+ 3us) write_op.cc:312] APPLY: finished
0804 14:52:13.086648 (+ 79us) log.cc:842] Serialized 22 byte log entry
0804 14:52:13.086749 (+ 101us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:13.086768 (+ 19us) write_op.cc:454] Released schema lock
0804 14:52:13.086775 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":256,"mutex_wait_us":17,"num_ops":1,"prepare.queue_time_us":517562,"prepare.run_cpu_time_us":752,"prepare.run_wall_time_us":285831,"raft.queue_time_us":131,"raft.run_cpu_time_us":419,"raft.run_wall_time_us":562,"replication_time_us":555356,"spinlock_wait_cycles":40576,"thread_start_us":108,"threads_started":1}]]}
I20260804 14:52:13.396701 28562 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:13.396875 28562 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:13.397583 28562 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:13.398079 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:13.398088 27988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:13.398633 28253 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:13.687443 28562 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:13.729652 28614 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=22, attempt_no=0}) took 1409 ms. Trace:
I20260804 14:52:13.729800 28614 rpcz_store.cc:276] 0804 14:52:12.320131 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:12.320229 (+ 98us) service_pool.cc:227] Handling call
0804 14:52:13.729626 (+1409397us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:12.530998 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:12.531114 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:12.531119 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:52:12.531122 (+ 3us) tablet.cc:660] Decoding operations
0804 14:52:12.531232 (+ 110us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:12.531244 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:12.531250 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:12.531305 (+ 55us) tablet.cc:699] Row locks acquired
0804 14:52:12.531307 (+ 2us) write_op.cc:260] PREPARE: finished
0804 14:52:12.531350 (+ 43us) write_op.cc:270] Start()
0804 14:52:12.844563 (+313213us) write_op.cc:276] Timestamp: P: 1785855132531334 usec, L: 0
0804 14:52:12.844568 (+ 5us) op_driver.cc:331] REPLICATION: starting
0804 14:52:12.844869 (+ 301us) log.cc:842] Serialized 354 byte log entry
0804 14:52:13.728819 (+883950us) op_driver.cc:444] REPLICATION: finished
0804 14:52:13.729106 (+ 287us) write_op.cc:301] APPLY: starting
0804 14:52:13.729131 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:13.729168 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:13.729186 (+ 18us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:13.729254 (+ 68us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:13.729308 (+ 54us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:13.729314 (+ 6us) write_op.cc:312] APPLY: finished
0804 14:52:13.729415 (+ 101us) log.cc:842] Serialized 22 byte log entry
0804 14:52:13.729499 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:13.729513 (+ 14us) write_op.cc:454] Released schema lock
0804 14:52:13.729518 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":223,"num_ops":1,"prepare.queue_time_us":210352,"prepare.run_cpu_time_us":885,"prepare.run_wall_time_us":313967,"replication_time_us":884219,"thread_start_us":109,"threads_started":1}]]}
I20260804 14:52:13.811676 28595 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260804 14:52:14.000121 28283 tablet_service.cc:2970] error setting up scanner 435edf8d404842559f116d4abd507255 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.311s (maximum allowed 0.100s): new_scan_request { tablet_id: "6cd92ff096234dafa214acc72d617f73" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\273\373\307\323\006\"3\n\005slave\022*\n 41255276b38541d68bbfe0d0057a0118\020\001\030\001 \001(\001" signature: "<redacted>"")\215P\256\002\302\361\025)\236\rI]\255\241^\250\030+\004\3636!j\276lc$\025\300l\027N\275O\035T\274\302\3555\200\216\322\212\242\210\302[\342r\211s\325PR\200\310M\227\201x\204\354" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "a778d080873b4800b57ddf4a1fab5db6"
W20260804 14:52:14.126868 28283 tablet_service.cc:2970] error setting up scanner a65fc709a4e643ddbcdcc04ad113d202 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.437s (maximum allowed 0.100s): new_scan_request { tablet_id: "6cd92ff096234dafa214acc72d617f73" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\273\373\307\323\006\"3\n\005slave\022*\n 41255276b38541d68bbfe0d0057a0118\020\001\030\001 \001(\001" signature: "<redacted>"")\215P\256\002\302\361\025)\236\rI]\255\241^\250\030+\004\3636!j\276lc$\025\300l\027N\275O\035T\274\302\3555\200\216\322\212\242\210\302[\342r\211s\325PR\200\310M\227\201x\204\354" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "f38c4acc68bb473786d01674d0ae9341"
I20260804 14:52:14.379884 28562 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:14.380035 28562 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:14.380654 28562 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:14.381038 27988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:14.381100 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:14.381525 28252 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:14.381724 28562 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:14.406827 28622 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=23, attempt_no=0}) took 1318 ms. Trace:
I20260804 14:52:14.406955 28622 rpcz_store.cc:276] 0804 14:52:13.088802 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:13.088873 (+ 71us) service_pool.cc:227] Handling call
0804 14:52:14.406790 (+1317917us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:13.089331 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:13.089424 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:13.089427 (+ 3us) write_op.cc:435] Acquired schema lock
0804 14:52:13.089428 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:13.089491 (+ 63us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:13.089498 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:13.089500 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:13.089531 (+ 31us) tablet.cc:699] Row locks acquired
0804 14:52:13.089532 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:13.089563 (+ 31us) write_op.cc:270] Start()
0804 14:52:14.021764 (+932201us) write_op.cc:276] Timestamp: P: 1785855133089508 usec, L: 0
0804 14:52:14.021768 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:52:14.021937 (+ 169us) log.cc:842] Serialized 354 byte log entry
0804 14:52:14.405840 (+383903us) op_driver.cc:444] REPLICATION: finished
0804 14:52:14.406178 (+ 338us) write_op.cc:301] APPLY: starting
0804 14:52:14.406205 (+ 27us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:14.406287 (+ 82us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:14.406320 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:14.406419 (+ 99us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:14.406450 (+ 31us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:14.406454 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:52:14.406540 (+ 86us) log.cc:842] Serialized 22 byte log entry
0804 14:52:14.406663 (+ 123us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:14.406679 (+ 16us) write_op.cc:454] Released schema lock
0804 14:52:14.406685 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":262,"num_ops":1,"prepare.queue_time_us":57,"prepare.run_cpu_time_us":631,"prepare.run_wall_time_us":932708,"raft.queue_time_us":55,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":232,"replication_time_us":384026,"thread_start_us":139,"threads_started":1}]]}
I20260804 14:52:14.921211 28595 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:14.921418 28595 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:14.922154 28595 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:52:14.922524 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:14.922887 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:52:14.923346 27893 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:15.292465 28562 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:15.292649 28562 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:15.293246 28562 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:15.293687 27988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:15.293789 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:15.294179 28252 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:15.328841 28632 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=24, attempt_no=0}) took 1597 ms. Trace:
I20260804 14:52:15.328954 28632 rpcz_store.cc:276] 0804 14:52:13.731596 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:13.731671 (+ 75us) service_pool.cc:227] Handling call
0804 14:52:15.328809 (+1597138us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:14.022067 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:14.022212 (+ 145us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:14.022217 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:52:14.022219 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:14.022300 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:14.022314 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:14.022317 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:14.022365 (+ 48us) tablet.cc:699] Row locks acquired
0804 14:52:14.022365 (+ 0us) write_op.cc:260] PREPARE: finished
0804 14:52:14.022403 (+ 38us) write_op.cc:270] Start()
0804 14:52:14.156619 (+134216us) write_op.cc:276] Timestamp: P: 1785855134022367 usec, L: 0
0804 14:52:14.156625 (+ 6us) op_driver.cc:331] REPLICATION: starting
0804 14:52:14.156857 (+ 232us) log.cc:842] Serialized 354 byte log entry
0804 14:52:15.327728 (+1170871us) op_driver.cc:444] REPLICATION: finished
0804 14:52:15.328088 (+ 360us) write_op.cc:301] APPLY: starting
0804 14:52:15.328115 (+ 27us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:15.328156 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:15.328176 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:15.328260 (+ 84us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:15.328289 (+ 29us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:15.328292 (+ 3us) write_op.cc:312] APPLY: finished
0804 14:52:15.328380 (+ 88us) log.cc:842] Serialized 22 byte log entry
0804 14:52:15.328668 (+ 288us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:15.328685 (+ 17us) write_op.cc:454] Released schema lock
0804 14:52:15.328694 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":287,"num_ops":1,"prepare.queue_time_us":289987,"prepare.run_cpu_time_us":810,"prepare.run_wall_time_us":134870,"replication_time_us":1171045,"thread_start_us":125,"threads_started":1}]]}
I20260804 14:52:15.438652 28595 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:15.512405 28562 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:15.559350 28632 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=25, attempt_no=0}) took 1150 ms. Trace:
I20260804 14:52:15.559537 28632 rpcz_store.cc:276] 0804 14:52:14.408532 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:14.408636 (+ 104us) service_pool.cc:227] Handling call
0804 14:52:15.559334 (+1150698us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:14.409105 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:14.409210 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:14.409214 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:52:14.409215 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:14.409286 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:14.409294 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:14.409296 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:14.409328 (+ 32us) tablet.cc:699] Row locks acquired
0804 14:52:14.409329 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:14.409363 (+ 34us) write_op.cc:270] Start()
0804 14:52:15.118535 (+709172us) write_op.cc:276] Timestamp: P: 1785855134409292 usec, L: 0
0804 14:52:15.118538 (+ 3us) op_driver.cc:331] REPLICATION: starting
0804 14:52:15.118809 (+ 271us) log.cc:842] Serialized 354 byte log entry
0804 14:52:15.558665 (+439856us) op_driver.cc:444] REPLICATION: finished
0804 14:52:15.558795 (+ 130us) write_op.cc:301] APPLY: starting
0804 14:52:15.558816 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:15.558848 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:15.558874 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:15.558965 (+ 91us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:15.558995 (+ 30us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:15.558998 (+ 3us) write_op.cc:312] APPLY: finished
0804 14:52:15.559132 (+ 134us) log.cc:842] Serialized 22 byte log entry
0804 14:52:15.559229 (+ 97us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:15.559247 (+ 18us) write_op.cc:454] Released schema lock
0804 14:52:15.559256 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":53,"num_ops":1,"prepare.queue_time_us":52,"prepare.run_cpu_time_us":728,"prepare.run_wall_time_us":709796,"replication_time_us":440092}]]}
I20260804 14:52:15.962460 28595 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:15.962622 28595 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:15.963253 28595 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:52:15.963665 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:15.963762 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:52:15.964323 27893 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:15.964483 28595 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:16.349682 28562 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:16.349836 28562 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:16.350387 28562 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:16.350826 27988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:16.351038 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:16.351439 28252 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:16.356479 28645 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=26, attempt_no=0}) took 1026 ms. Trace:
I20260804 14:52:16.356616 28645 rpcz_store.cc:276] 0804 14:52:15.330446 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:15.330522 (+ 76us) service_pool.cc:227] Handling call
0804 14:52:16.356460 (+1025938us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:15.330976 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:15.331069 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:15.331072 (+ 3us) write_op.cc:435] Acquired schema lock
0804 14:52:15.331073 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:15.331136 (+ 63us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:15.331143 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:15.331145 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:15.331177 (+ 32us) tablet.cc:699] Row locks acquired
0804 14:52:15.331178 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:15.331266 (+ 88us) write_op.cc:270] Start()
0804 14:52:16.002452 (+671186us) write_op.cc:276] Timestamp: P: 1785855135331204 usec, L: 0
0804 14:52:16.002456 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:52:16.002634 (+ 178us) log.cc:842] Serialized 354 byte log entry
0804 14:52:16.355744 (+353110us) op_driver.cc:444] REPLICATION: finished
0804 14:52:16.356062 (+ 318us) write_op.cc:301] APPLY: starting
0804 14:52:16.356083 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:16.356130 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:16.356145 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:16.356204 (+ 59us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:16.356230 (+ 26us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:16.356233 (+ 3us) write_op.cc:312] APPLY: finished
0804 14:52:16.356298 (+ 65us) log.cc:842] Serialized 22 byte log entry
0804 14:52:16.356373 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:16.356384 (+ 11us) write_op.cc:454] Released schema lock
0804 14:52:16.356389 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":265,"num_ops":1,"prepare.queue_time_us":65,"prepare.run_cpu_time_us":696,"prepare.run_wall_time_us":671772,"raft.queue_time_us":33,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":191,"replication_time_us":353257,"spinlock_wait_cycles":896,"thread_start_us":180,"threads_started":1}]]}
I20260804 14:52:16.406556 28562 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:16.567727 28595 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:16.567874 28595 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:16.568394 28595 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:52:16.568746 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:16.568781 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:52:16.569435 27891 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:16.760996 28595 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:16.900561 28595 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:16.900724 28595 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:16.901269 28595 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:52:16.901719 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:52:16.901743 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:16.902200 27891 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:16.902412 28595 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:17.399231 28595 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:17.399410 28595 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:17.399961 28595 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:52:17.400413 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:52:17.400410 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:17.400954 27893 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:17.401140 28595 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:17.617846 28562 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:17.618028 28562 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:17.618677 28562 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:17.619095 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:17.619380 27988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:17.619850 28253 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:18.086874 28595 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:18.087101 28595 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:18.087774 28595 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:52:18.088167 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:18.088279 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:52:18.088696 27893 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:18.109081 28663 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=30, attempt_no=0}) took 1192 ms. Trace:
I20260804 14:52:18.109184 28663 rpcz_store.cc:276] 0804 14:52:16.916752 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:16.916814 (+ 62us) service_pool.cc:227] Handling call
0804 14:52:18.109058 (+1192244us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:16.917559 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:16.917689 (+ 130us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:16.917692 (+ 3us) write_op.cc:435] Acquired schema lock
0804 14:52:16.917694 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:16.917760 (+ 66us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:16.917767 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:16.917769 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:16.917801 (+ 32us) tablet.cc:699] Row locks acquired
0804 14:52:16.917802 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:16.917835 (+ 33us) write_op.cc:270] Start()
0804 14:52:17.430973 (+513138us) write_op.cc:276] Timestamp: P: 1785855136917785 usec, L: 0
0804 14:52:17.430976 (+ 3us) op_driver.cc:331] REPLICATION: starting
0804 14:52:17.431172 (+ 196us) log.cc:842] Serialized 354 byte log entry
0804 14:52:18.108173 (+677001us) op_driver.cc:444] REPLICATION: finished
0804 14:52:18.108560 (+ 387us) write_op.cc:301] APPLY: starting
0804 14:52:18.108593 (+ 33us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:18.108629 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:18.108650 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:18.108721 (+ 71us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:18.108749 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:18.108752 (+ 3us) write_op.cc:312] APPLY: finished
0804 14:52:18.108834 (+ 82us) log.cc:842] Serialized 22 byte log entry
0804 14:52:18.108955 (+ 121us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:18.108968 (+ 13us) write_op.cc:454] Released schema lock
0804 14:52:18.108973 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":249,"num_ops":1,"prepare.queue_time_us":52,"prepare.run_cpu_time_us":679,"prepare.run_wall_time_us":513744,"replication_time_us":677148,"spinlock_wait_cycles":8576,"thread_start_us":92,"threads_started":1}]]}
W20260804 14:52:18.481397 28283 tablet_service.cc:2970] error setting up scanner c5a31445e295457cad8e7d81bafa360f with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.862s (maximum allowed 0.100s): new_scan_request { tablet_id: "6cd92ff096234dafa214acc72d617f73" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\273\373\307\323\006\"3\n\005slave\022*\n 41255276b38541d68bbfe0d0057a0118\020\001\030\001 \001(\001" signature: "<redacted>"")\215P\256\002\302\361\025)\236\rI]\255\241^\250\030+\004\3636!j\276lc$\025\300l\027N\275O\035T\274\302\3555\200\216\322\212\242\210\302[\342r\211s\325PR\200\310M\227\201x\204\354" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "f38c4acc68bb473786d01674d0ae9341"
W20260804 14:52:18.610534 28283 tablet_service.cc:2970] error setting up scanner eefc42ee77b34660bc19e7d1e8b13147 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.991s (maximum allowed 0.100s): new_scan_request { tablet_id: "6cd92ff096234dafa214acc72d617f73" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\273\373\307\323\006\"3\n\005slave\022*\n 41255276b38541d68bbfe0d0057a0118\020\001\030\001 \001(\001" signature: "<redacted>"")\215P\256\002\302\361\025)\236\rI]\255\241^\250\030+\004\3636!j\276lc$\025\300l\027N\275O\035T\274\302\3555\200\216\322\212\242\210\302[\342r\211s\325PR\200\310M\227\201x\204\354" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "a778d080873b4800b57ddf4a1fab5db6"
I20260804 14:52:18.762184 28595 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:18.784574 28671 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=31, attempt_no=0}) took 1381 ms. Trace:
I20260804 14:52:18.784691 28671 rpcz_store.cc:276] 0804 14:52:17.403322 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:17.403367 (+ 45us) service_pool.cc:227] Handling call
0804 14:52:18.784529 (+1381162us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:17.431315 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:17.431444 (+ 129us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:17.431448 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:52:17.431449 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:17.431543 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:17.431552 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:17.431555 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:17.431593 (+ 38us) tablet.cc:699] Row locks acquired
0804 14:52:17.431594 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:17.431631 (+ 37us) write_op.cc:270] Start()
0804 14:52:17.929789 (+498158us) write_op.cc:276] Timestamp: P: 1785855137431573 usec, L: 0
0804 14:52:17.929794 (+ 5us) op_driver.cc:331] REPLICATION: starting
0804 14:52:17.930041 (+ 247us) log.cc:842] Serialized 354 byte log entry
0804 14:52:18.783572 (+853531us) op_driver.cc:444] REPLICATION: finished
0804 14:52:18.783922 (+ 350us) write_op.cc:301] APPLY: starting
0804 14:52:18.783957 (+ 35us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:18.783990 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:18.784010 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:18.784077 (+ 67us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:18.784105 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:18.784108 (+ 3us) write_op.cc:312] APPLY: finished
0804 14:52:18.784266 (+ 158us) log.cc:842] Serialized 22 byte log entry
0804 14:52:18.784381 (+ 115us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:18.784398 (+ 17us) write_op.cc:454] Released schema lock
0804 14:52:18.784404 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":258,"num_ops":1,"prepare.queue_time_us":27491,"prepare.run_cpu_time_us":739,"prepare.run_wall_time_us":498810,"replication_time_us":853701,"thread_start_us":105,"threads_started":1}]]}
I20260804 14:52:18.937852 28562 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:18.938235 28347 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.24.6.130:53843 (request call id 3645) took 1318 ms. Trace:
I20260804 14:52:18.938324 28347 rpcz_store.cc:276] 0804 14:52:17.619465 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:17.619516 (+ 51us) service_pool.cc:227] Handling call
0804 14:52:17.619662 (+ 146us) raft_consensus.cc:1457] Updating replica for 2 ops
0804 14:52:17.619762 (+ 100us) raft_consensus.cc:1520] Early marking committed up to index 30
0804 14:52:17.619931 (+ 169us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0804 14:52:17.620052 (+ 121us) write_op.cc:270] Start()
0804 14:52:17.956284 (+336232us) write_op.cc:276] Timestamp: P: 1785855136684503 usec, L: 0
0804 14:52:17.956762 (+ 478us) write_op.cc:270] Start()
0804 14:52:18.936926 (+980164us) write_op.cc:276] Timestamp: P: 1785855136917785 usec, L: 0
0804 14:52:18.937410 (+ 484us) log.cc:842] Serialized 708 byte log entry
0804 14:52:18.937610 (+ 200us) raft_consensus.cc:1631] Marking committed up to 31
0804 14:52:18.937755 (+ 145us) raft_consensus.cc:1683] Filling consensus response to leader.
0804 14:52:18.937792 (+ 37us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0804 14:52:18.938169 (+ 377us) raft_consensus.cc:1671] finished
0804 14:52:18.938171 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0804 14:52:18.938218 (+ 47us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:17.956592 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:17.956704 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:17.956708 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:52:17.956710 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:17.956797 (+ 87us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:17.956809 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:17.956812 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:17.956853 (+ 41us) tablet.cc:699] Row locks acquired
0804 14:52:17.956854 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:18.937646 (+980792us) op_driver.cc:444] REPLICATION: finished
0804 14:52:18.937857 (+ 211us) write_op.cc:301] APPLY: starting
0804 14:52:18.937876 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:18.937914 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:18.937915 (+ 1us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:18.937978 (+ 63us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:18.938009 (+ 31us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:18.938032 (+ 23us) write_op.cc:312] APPLY: finished
0804 14:52:18.938109 (+ 77us) log.cc:842] Serialized 22 byte log entry
0804 14:52:18.938198 (+ 89us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:18.938207 (+ 9us) write_op.cc:454] Released schema lock
0804 14:52:18.938211 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0804 14:52:18.938264 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:18.938265 (+ 1us) write_op.cc:454] Released schema lock
Related trace 'op':
0804 14:52:18.937307 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:18.937415 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:18.937419 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:52:18.937421 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:18.937489 (+ 68us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:18.937523 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:18.937525 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:18.937563 (+ 38us) tablet.cc:699] Row locks acquired
0804 14:52:18.937564 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":138,"threads_started":1,"wal-append.queue_time_us":206,"child_traces":[["op",{"apply.queue_time_us":160,"apply.run_cpu_time_us":445,"apply.run_wall_time_us":445,"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":115,"prepare.run_cpu_time_us":379,"prepare.run_wall_time_us":370,"replication_time_us":1317580,"thread_start_us":61,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":29,"prepare.queue_time_us":239,"prepare.run_cpu_time_us":328,"prepare.run_wall_time_us":329,"thread_start_us":95,"threads_started":1}]]}
I20260804 14:52:19.072064 28595 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:19.072252 28595 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:19.072902 28595 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:52:19.073266 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:19.073446 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:52:19.073846 27893 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:19.074059 28595 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:20.304319 28676 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:20.304481 28676 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:20.305024 28676 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:52:20.305461 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:20.305560 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:52:20.306015 27893 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:20.306242 28676 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:20.333834 28684 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=33, attempt_no=0}) took 1547 ms. Trace:
I20260804 14:52:20.333945 28684 rpcz_store.cc:276] 0804 14:52:18.786236 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:18.786317 (+ 81us) service_pool.cc:227] Handling call
0804 14:52:20.333777 (+1547460us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:18.786809 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:18.786900 (+ 91us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:18.786903 (+ 3us) write_op.cc:435] Acquired schema lock
0804 14:52:18.786904 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:18.786972 (+ 68us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:18.786980 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:18.786983 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:18.787018 (+ 35us) tablet.cc:699] Row locks acquired
0804 14:52:18.787018 (+ 0us) write_op.cc:260] PREPARE: finished
0804 14:52:18.787049 (+ 31us) write_op.cc:270] Start()
0804 14:52:19.713253 (+926204us) write_op.cc:276] Timestamp: P: 1785855138787023 usec, L: 0
0804 14:52:19.713257 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:52:19.713447 (+ 190us) log.cc:842] Serialized 354 byte log entry
0804 14:52:20.332823 (+619376us) op_driver.cc:444] REPLICATION: finished
0804 14:52:20.333250 (+ 427us) write_op.cc:301] APPLY: starting
0804 14:52:20.333279 (+ 29us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:20.333326 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:20.333347 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:20.333441 (+ 94us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:20.333473 (+ 32us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:20.333477 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:52:20.333557 (+ 80us) log.cc:842] Serialized 22 byte log entry
0804 14:52:20.333626 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:20.333641 (+ 15us) write_op.cc:454] Released schema lock
0804 14:52:20.333646 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":339,"num_ops":1,"prepare.queue_time_us":55,"prepare.run_cpu_time_us":616,"prepare.run_wall_time_us":926721,"replication_time_us":619515,"thread_start_us":159,"threads_started":1}]]}
I20260804 14:52:20.628624 28562 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:20.628791 28562 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:20.628971 28347 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.24.6.130:53843 (request call id 3646) took 1688 ms. Trace:
I20260804 14:52:20.629055 28347 rpcz_store.cc:276] 0804 14:52:18.940110 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:18.940194 (+ 84us) service_pool.cc:227] Handling call
0804 14:52:18.940353 (+ 159us) raft_consensus.cc:1457] Updating replica for 2 ops
0804 14:52:18.940421 (+ 68us) raft_consensus.cc:1520] Early marking committed up to index 32
0804 14:52:18.940489 (+ 68us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0804 14:52:18.940551 (+ 62us) write_op.cc:270] Start()
0804 14:52:19.855721 (+915170us) write_op.cc:276] Timestamp: P: 1785855137431573 usec, L: 0
0804 14:52:19.856205 (+ 484us) write_op.cc:270] Start()
0804 14:52:20.627463 (+771258us) write_op.cc:276] Timestamp: P: 1785855138111487 usec, L: 0
0804 14:52:20.628173 (+ 710us) log.cc:842] Serialized 708 byte log entry
0804 14:52:20.628339 (+ 166us) raft_consensus.cc:1631] Marking committed up to 33
0804 14:52:20.628516 (+ 177us) raft_consensus.cc:1683] Filling consensus response to leader.
0804 14:52:20.628556 (+ 40us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0804 14:52:20.628893 (+ 337us) raft_consensus.cc:1671] finished
0804 14:52:20.628894 (+ 1us) raft_consensus.cc:1677] UpdateReplicas() finished
0804 14:52:20.628950 (+ 56us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:19.857398 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:19.857642 (+ 244us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:19.857649 (+ 7us) write_op.cc:435] Acquired schema lock
0804 14:52:19.857653 (+ 4us) tablet.cc:660] Decoding operations
0804 14:52:19.857767 (+ 114us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:19.857788 (+ 21us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:19.857793 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:19.857855 (+ 62us) tablet.cc:699] Row locks acquired
0804 14:52:19.857858 (+ 3us) write_op.cc:260] PREPARE: finished
0804 14:52:20.628388 (+770530us) op_driver.cc:444] REPLICATION: finished
0804 14:52:20.628680 (+ 292us) write_op.cc:301] APPLY: starting
0804 14:52:20.628701 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:20.628738 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:20.628742 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:20.628807 (+ 65us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:20.628839 (+ 32us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:20.628894 (+ 55us) write_op.cc:312] APPLY: finished
0804 14:52:20.628964 (+ 70us) log.cc:842] Serialized 22 byte log entry
0804 14:52:20.629027 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:20.629047 (+ 20us) write_op.cc:454] Released schema lock
0804 14:52:20.629052 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0804 14:52:20.629127 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:20.629131 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0804 14:52:20.628023 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:20.628139 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:20.628143 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:52:20.628145 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:20.628220 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:20.628232 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:20.628234 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:20.628275 (+ 41us) tablet.cc:699] Row locks acquired
0804 14:52:20.628276 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":98,"threads_started":1,"wal-append.queue_time_us":169,"child_traces":[["op",{"apply.queue_time_us":226,"apply.run_cpu_time_us":500,"apply.run_wall_time_us":499,"num_ops":1,"partition_lock_wait_us":11,"prepare.queue_time_us":1511,"prepare.run_cpu_time_us":581,"prepare.run_wall_time_us":580,"replication_time_us":1687820,"thread_start_us":197,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":405,"prepare.run_cpu_time_us":336,"prepare.run_wall_time_us":337,"thread_start_us":176,"threads_started":1}]]}
I20260804 14:52:20.629463 28562 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:20.630038 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:20.630048 27988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:20.630746 28253 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:21.089862 28676 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:21.090060 28676 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:21.090879 28676 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:52:21.091343 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:21.091437 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:52:21.092334 27893 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:21.121778 28693 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=34, attempt_no=0}) took 2035 ms. Trace:
I20260804 14:52:21.121964 28693 rpcz_store.cc:276] 0804 14:52:19.086452 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:19.086500 (+ 48us) service_pool.cc:227] Handling call
0804 14:52:21.121749 (+2035249us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:19.713589 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:19.713705 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:19.713713 (+ 8us) write_op.cc:435] Acquired schema lock
0804 14:52:19.713715 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:19.713801 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:19.713813 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:19.713816 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:19.713860 (+ 44us) tablet.cc:699] Row locks acquired
0804 14:52:19.713861 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:19.713910 (+ 49us) write_op.cc:270] Start()
0804 14:52:19.907666 (+193756us) write_op.cc:276] Timestamp: P: 1785855139713881 usec, L: 0
0804 14:52:19.907671 (+ 5us) op_driver.cc:331] REPLICATION: starting
0804 14:52:19.907897 (+ 226us) log.cc:842] Serialized 354 byte log entry
0804 14:52:21.120574 (+1212677us) op_driver.cc:444] REPLICATION: finished
0804 14:52:21.121117 (+ 543us) write_op.cc:301] APPLY: starting
0804 14:52:21.121146 (+ 29us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:21.121183 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:21.121205 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:21.121284 (+ 79us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:21.121316 (+ 32us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:21.121319 (+ 3us) write_op.cc:312] APPLY: finished
0804 14:52:21.121485 (+ 166us) log.cc:842] Serialized 22 byte log entry
0804 14:52:21.121600 (+ 115us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:21.121620 (+ 20us) write_op.cc:454] Released schema lock
0804 14:52:21.121627 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":429,"num_ops":1,"prepare.queue_time_us":626652,"prepare.run_cpu_time_us":1649,"prepare.run_wall_time_us":194374,"replication_time_us":1212824,"thread_start_us":236,"threads_started":1}]]}
W20260804 14:52:21.299971 28283 tablet_service.cc:2970] error setting up scanner f757dda7f80b4ed198bd01cb3ae22900 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.669s (maximum allowed 0.100s): new_scan_request { tablet_id: "6cd92ff096234dafa214acc72d617f73" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\273\373\307\323\006\"3\n\005slave\022*\n 41255276b38541d68bbfe0d0057a0118\020\001\030\001 \001(\001" signature: "<redacted>"")\215P\256\002\302\361\025)\236\rI]\255\241^\250\030+\004\3636!j\276lc$\025\300l\027N\275O\035T\274\302\3555\200\216\322\212\242\210\302[\342r\211s\325PR\200\310M\227\201x\204\354" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "f38c4acc68bb473786d01674d0ae9341"
I20260804 14:52:21.551677 28676 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:21.576655 28693 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=35, attempt_no=0}) took 1241 ms. Trace:
I20260804 14:52:21.576777 28693 rpcz_store.cc:276] 0804 14:52:20.335565 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:20.335627 (+ 62us) service_pool.cc:227] Handling call
0804 14:52:21.576635 (+1241008us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:20.336146 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:20.336243 (+ 97us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:20.336246 (+ 3us) write_op.cc:435] Acquired schema lock
0804 14:52:20.336248 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:20.336321 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:20.336330 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:20.336333 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:20.336368 (+ 35us) tablet.cc:699] Row locks acquired
0804 14:52:20.336369 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:20.336409 (+ 40us) write_op.cc:270] Start()
0804 14:52:20.344580 (+ 8171us) write_op.cc:276] Timestamp: P: 1785855140336324 usec, L: 0
0804 14:52:20.344585 (+ 5us) op_driver.cc:331] REPLICATION: starting
0804 14:52:20.344860 (+ 275us) log.cc:842] Serialized 354 byte log entry
0804 14:52:21.575767 (+1230907us) op_driver.cc:444] REPLICATION: finished
0804 14:52:21.575986 (+ 219us) write_op.cc:301] APPLY: starting
0804 14:52:21.576014 (+ 28us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:21.576048 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:21.576070 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:21.576141 (+ 71us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:21.576289 (+ 148us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:21.576297 (+ 8us) write_op.cc:312] APPLY: finished
0804 14:52:21.576416 (+ 119us) log.cc:842] Serialized 22 byte log entry
0804 14:52:21.576531 (+ 115us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:21.576545 (+ 14us) write_op.cc:454] Released schema lock
0804 14:52:21.576551 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":156,"num_ops":1,"prepare.queue_time_us":79,"prepare.run_cpu_time_us":757,"prepare.run_wall_time_us":8826,"replication_time_us":1231128}]]}
I20260804 14:52:21.596343 28562 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260804 14:52:21.942687 28283 tablet_service.cc:2970] error setting up scanner 689cdfb57f324fae891a81f062e5beb7 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.344s (maximum allowed 0.100s): new_scan_request { tablet_id: "6cd92ff096234dafa214acc72d617f73" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\273\373\307\323\006\"3\n\005slave\022*\n 41255276b38541d68bbfe0d0057a0118\020\001\030\001 \001(\001" signature: "<redacted>"")\215P\256\002\302\361\025)\236\rI]\255\241^\250\030+\004\3636!j\276lc$\025\300l\027N\275O\035T\274\302\3555\200\216\322\212\242\210\302[\342r\211s\325PR\200\310M\227\201x\204\354" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "a778d080873b4800b57ddf4a1fab5db6"
I20260804 14:52:22.213055 28676 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:22.213238 28676 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:22.213882 28676 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:52:22.214318 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:22.214383 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:52:22.214996 27893 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:22.233022 28562 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:22.233184 28562 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:22.233733 28562 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:22.233721 28708 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=36, attempt_no=0}) took 1109 ms. Trace:
I20260804 14:52:22.233801 28708 rpcz_store.cc:276] 0804 14:52:21.124343 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:21.124481 (+ 138us) service_pool.cc:227] Handling call
0804 14:52:22.233699 (+1109218us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:21.125440 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:21.125536 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:21.125540 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:52:21.125541 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:21.125611 (+ 70us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:21.125620 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:21.125621 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:21.125665 (+ 44us) tablet.cc:699] Row locks acquired
0804 14:52:21.125666 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:21.125702 (+ 36us) write_op.cc:270] Start()
0804 14:52:21.526929 (+401227us) write_op.cc:276] Timestamp: P: 1785855141125615 usec, L: 0
0804 14:52:21.526934 (+ 5us) op_driver.cc:331] REPLICATION: starting
0804 14:52:21.527234 (+ 300us) log.cc:842] Serialized 354 byte log entry
0804 14:52:22.232659 (+705425us) op_driver.cc:444] REPLICATION: finished
0804 14:52:22.233014 (+ 355us) write_op.cc:301] APPLY: starting
0804 14:52:22.233033 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:22.233069 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:22.233091 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:22.233218 (+ 127us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:22.233252 (+ 34us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:22.233257 (+ 5us) write_op.cc:312] APPLY: finished
0804 14:52:22.233339 (+ 82us) log.cc:842] Serialized 22 byte log entry
0804 14:52:22.233584 (+ 245us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:22.233594 (+ 10us) write_op.cc:454] Released schema lock
0804 14:52:22.233599 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":287,"num_ops":1,"prepare.queue_time_us":377,"prepare.run_cpu_time_us":779,"prepare.run_wall_time_us":401878,"replication_time_us":705693,"thread_start_us":413,"threads_started":2}]]}
I20260804 14:52:22.234211 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:22.234256 27988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:22.234818 28253 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:22.617396 28562 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:22.650271 28708 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=37, attempt_no=0}) took 1072 ms. Trace:
I20260804 14:52:22.650398 28708 rpcz_store.cc:276] 0804 14:52:21.578298 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:21.578361 (+ 63us) service_pool.cc:227] Handling call
0804 14:52:22.650233 (+1071872us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:21.578816 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:21.578905 (+ 89us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:21.578908 (+ 3us) write_op.cc:435] Acquired schema lock
0804 14:52:21.578909 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:21.578973 (+ 64us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:21.578980 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:21.578982 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:21.579015 (+ 33us) tablet.cc:699] Row locks acquired
0804 14:52:21.579015 (+ 0us) write_op.cc:260] PREPARE: finished
0804 14:52:21.579047 (+ 32us) write_op.cc:270] Start()
0804 14:52:22.011291 (+432244us) write_op.cc:276] Timestamp: P: 1785855141578962 usec, L: 0
0804 14:52:22.011296 (+ 5us) op_driver.cc:331] REPLICATION: starting
0804 14:52:22.011725 (+ 429us) log.cc:842] Serialized 354 byte log entry
0804 14:52:22.649620 (+637895us) op_driver.cc:444] REPLICATION: finished
0804 14:52:22.649753 (+ 133us) write_op.cc:301] APPLY: starting
0804 14:52:22.649780 (+ 27us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:22.649818 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:22.649836 (+ 18us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:22.649932 (+ 96us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:22.649968 (+ 36us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:22.649972 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:52:22.650063 (+ 91us) log.cc:842] Serialized 22 byte log entry
0804 14:52:22.650147 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:22.650160 (+ 13us) write_op.cc:454] Released schema lock
0804 14:52:22.650165 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":59,"num_ops":1,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":883,"prepare.run_wall_time_us":432984,"replication_time_us":638274}]]}
I20260804 14:52:22.657562 28676 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:22.935792 28676 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:22.935971 28676 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:22.936605 28676 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:52:22.937026 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:22.937062 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:52:22.937763 27893 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:22.937969 28676 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:23.289295 28562 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:23.289462 28562 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:23.290010 28562 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:23.290316 27988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:23.290444 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:23.290966 28253 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:23.291301 28562 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260804 14:52:24.118497 28283 tablet_service.cc:2970] error setting up scanner 746777e82f024bd6a8038a2c7be3a701 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.828s (maximum allowed 0.100s): new_scan_request { tablet_id: "6cd92ff096234dafa214acc72d617f73" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\273\373\307\323\006\"3\n\005slave\022*\n 41255276b38541d68bbfe0d0057a0118\020\001\030\001 \001(\001" signature: "<redacted>"")\215P\256\002\302\361\025)\236\rI]\255\241^\250\030+\004\3636!j\276lc$\025\300l\027N\275O\035T\274\302\3555\200\216\322\212\242\210\302[\342r\211s\325PR\200\310M\227\201x\204\354" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "f38c4acc68bb473786d01674d0ae9341"
I20260804 14:52:24.466909 28714 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:24.467078 28714 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:24.467908 28714 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:24.468160 27989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:24.468195 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:24.468710 28252 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:24.469118 28714 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:24.481380 28723 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=39, attempt_no=0}) took 1829 ms. Trace:
I20260804 14:52:24.481528 28723 rpcz_store.cc:276] 0804 14:52:22.651922 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:22.651963 (+ 41us) service_pool.cc:227] Handling call
0804 14:52:24.481359 (+1829396us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:22.914230 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:22.914337 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:22.914341 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:52:22.914342 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:22.914446 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:22.914457 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:22.914460 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:22.914516 (+ 56us) tablet.cc:699] Row locks acquired
0804 14:52:22.914518 (+ 2us) write_op.cc:260] PREPARE: finished
0804 14:52:22.914565 (+ 47us) write_op.cc:270] Start()
0804 14:52:23.853740 (+939175us) write_op.cc:276] Timestamp: P: 1785855142914502 usec, L: 0
0804 14:52:23.853744 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:52:23.853945 (+ 201us) log.cc:842] Serialized 354 byte log entry
0804 14:52:24.480682 (+626737us) op_driver.cc:444] REPLICATION: finished
0804 14:52:24.480925 (+ 243us) write_op.cc:301] APPLY: starting
0804 14:52:24.480985 (+ 60us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:24.481018 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:24.481032 (+ 14us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:24.481094 (+ 62us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:24.481121 (+ 27us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:24.481124 (+ 3us) write_op.cc:312] APPLY: finished
0804 14:52:24.481189 (+ 65us) log.cc:842] Serialized 22 byte log entry
0804 14:52:24.481239 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:24.481250 (+ 11us) write_op.cc:454] Released schema lock
0804 14:52:24.481254 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":182,"num_ops":1,"prepare.queue_time_us":261819,"prepare.run_cpu_time_us":716,"prepare.run_wall_time_us":939786,"replication_time_us":626898,"thread_start_us":77,"threads_started":1}]]}
I20260804 14:52:24.520004 28715 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:24.520156 28715 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:24.520771 28715 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:52:24.521181 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:52:24.521183 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:24.521785 27893 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:24.521982 28715 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:24.786873 28714 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:24.787039 28714 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:24.787710 28714 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:24.788241 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:24.788380 27989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:24.789417 28252 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:24.789660 28714 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:24.811901 28723 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=40, attempt_no=0}) took 1835 ms. Trace:
I20260804 14:52:24.812029 28723 rpcz_store.cc:276] 0804 14:52:22.976879 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:22.976961 (+ 82us) service_pool.cc:227] Handling call
0804 14:52:24.811883 (+1834922us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:23.854072 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:23.854192 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:23.854197 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:52:23.854199 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:23.854285 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:23.854295 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:23.854298 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:23.854342 (+ 44us) tablet.cc:699] Row locks acquired
0804 14:52:23.854344 (+ 2us) write_op.cc:260] PREPARE: finished
0804 14:52:23.854392 (+ 48us) write_op.cc:270] Start()
0804 14:52:24.548572 (+694180us) write_op.cc:276] Timestamp: P: 1785855143854331 usec, L: 0
0804 14:52:24.548576 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:52:24.548785 (+ 209us) log.cc:842] Serialized 354 byte log entry
0804 14:52:24.811285 (+262500us) op_driver.cc:444] REPLICATION: finished
0804 14:52:24.811404 (+ 119us) write_op.cc:301] APPLY: starting
0804 14:52:24.811432 (+ 28us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:24.811470 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:24.811485 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:24.811554 (+ 69us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:24.811583 (+ 29us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:24.811587 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:52:24.811658 (+ 71us) log.cc:842] Serialized 22 byte log entry
0804 14:52:24.811776 (+ 118us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:24.811791 (+ 15us) write_op.cc:454] Released schema lock
0804 14:52:24.811797 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":56,"mutex_wait_us":26,"num_ops":1,"prepare.queue_time_us":876653,"prepare.run_cpu_time_us":738,"prepare.run_wall_time_us":694823,"raft.queue_time_us":873,"raft.run_cpu_time_us":442,"raft.run_wall_time_us":471,"replication_time_us":262676}]]}
I20260804 14:52:25.098042 28715 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:25.098209 28715 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:25.098841 28715 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:52:25.099221 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:25.099236 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:52:25.099676 27893 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
W20260804 14:52:25.273881 28283 tablet_service.cc:2970] error setting up scanner c999fb1e330840159b39a09a39899012 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.485s (maximum allowed 0.100s): new_scan_request { tablet_id: "6cd92ff096234dafa214acc72d617f73" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\273\373\307\323\006\"3\n\005slave\022*\n 41255276b38541d68bbfe0d0057a0118\020\001\030\001 \001(\001" signature: "<redacted>"")\215P\256\002\302\361\025)\236\rI]\255\241^\250\030+\004\3636!j\276lc$\025\300l\027N\275O\035T\274\302\3555\200\216\322\212\242\210\302[\342r\211s\325PR\200\310M\227\201x\204\354" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "a778d080873b4800b57ddf4a1fab5db6"
I20260804 14:52:25.766628 28714 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:25.766798 28714 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:25.767406 28714 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:25.767829 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:25.767829 27989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:25.768366 28253 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:25.768565 28714 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:25.770378 28738 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=41, attempt_no=0}) took 1287 ms. Trace:
I20260804 14:52:25.770478 28738 rpcz_store.cc:276] 0804 14:52:24.482987 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:24.483057 (+ 70us) service_pool.cc:227] Handling call
0804 14:52:25.770364 (+1287307us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:24.548928 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:24.549055 (+ 127us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:24.549062 (+ 7us) write_op.cc:435] Acquired schema lock
0804 14:52:24.549064 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:24.549171 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:24.549183 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:24.549186 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:24.549237 (+ 51us) tablet.cc:699] Row locks acquired
0804 14:52:24.549239 (+ 2us) write_op.cc:260] PREPARE: finished
0804 14:52:24.549289 (+ 50us) write_op.cc:270] Start()
0804 14:52:25.024450 (+475161us) write_op.cc:276] Timestamp: P: 1785855144549255 usec, L: 0
0804 14:52:25.024477 (+ 27us) op_driver.cc:331] REPLICATION: starting
0804 14:52:25.024682 (+ 205us) log.cc:842] Serialized 354 byte log entry
0804 14:52:25.769864 (+745182us) op_driver.cc:444] REPLICATION: finished
0804 14:52:25.770076 (+ 212us) write_op.cc:301] APPLY: starting
0804 14:52:25.770089 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:25.770112 (+ 23us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:25.770126 (+ 14us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:25.770174 (+ 48us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:25.770196 (+ 22us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:25.770199 (+ 3us) write_op.cc:312] APPLY: finished
0804 14:52:25.770250 (+ 51us) log.cc:842] Serialized 22 byte log entry
0804 14:52:25.770296 (+ 46us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:25.770305 (+ 9us) write_op.cc:454] Released schema lock
0804 14:52:25.770309 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":163,"num_ops":1,"prepare.queue_time_us":65520,"prepare.run_cpu_time_us":756,"prepare.run_wall_time_us":475812,"replication_time_us":745361,"thread_start_us":69,"threads_started":1}]]}
I20260804 14:52:25.827239 28715 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:26.596976 28714 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:26.597151 28714 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:26.597779 28714 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:26.598229 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 44 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:26.598237 27989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 44 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:26.598735 28253 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:26.598901 28714 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:26.614487 28750 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=42, attempt_no=0}) took 1800 ms. Trace:
I20260804 14:52:26.614610 28750 rpcz_store.cc:276] 0804 14:52:24.813616 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:24.813688 (+ 72us) service_pool.cc:227] Handling call
0804 14:52:26.614464 (+1800776us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:25.024767 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:25.024892 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:25.024899 (+ 7us) write_op.cc:435] Acquired schema lock
0804 14:52:25.024901 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:25.024986 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:25.024996 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:25.025000 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:25.025066 (+ 66us) tablet.cc:699] Row locks acquired
0804 14:52:25.025067 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:25.025112 (+ 45us) write_op.cc:270] Start()
0804 14:52:25.450327 (+425215us) write_op.cc:276] Timestamp: P: 1785855145025081 usec, L: 0
0804 14:52:25.450332 (+ 5us) op_driver.cc:331] REPLICATION: starting
0804 14:52:25.450576 (+ 244us) log.cc:842] Serialized 354 byte log entry
0804 14:52:26.613683 (+1163107us) op_driver.cc:444] REPLICATION: finished
0804 14:52:26.613991 (+ 308us) write_op.cc:301] APPLY: starting
0804 14:52:26.614013 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:26.614045 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:26.614065 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:26.614132 (+ 67us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:26.614159 (+ 27us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:26.614162 (+ 3us) write_op.cc:312] APPLY: finished
0804 14:52:26.614238 (+ 76us) log.cc:842] Serialized 22 byte log entry
0804 14:52:26.614340 (+ 102us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:26.614357 (+ 17us) write_op.cc:454] Released schema lock
0804 14:52:26.614364 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":237,"num_ops":1,"prepare.queue_time_us":210722,"prepare.run_cpu_time_us":785,"prepare.run_wall_time_us":425871,"replication_time_us":1163297,"thread_start_us":107,"threads_started":1}]]}
I20260804 14:52:26.791450 28715 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:26.791597 28715 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:26.792131 28715 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:52:26.792514 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 44 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:26.792739 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 44 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:52:26.793229 27893 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:27.128286 28714 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:27.128489 28714 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:27.129123 28714 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:27.129577 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:27.129582 27989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:27.130125 28253 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:27.164659 28759 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=43, attempt_no=0}) took 1392 ms. Trace:
I20260804 14:52:27.164752 28759 rpcz_store.cc:276] 0804 14:52:25.772109 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:25.772188 (+ 79us) service_pool.cc:227] Handling call
0804 14:52:27.164640 (+1392452us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:25.772639 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:25.772721 (+ 82us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:25.772724 (+ 3us) write_op.cc:435] Acquired schema lock
0804 14:52:25.772725 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:25.772786 (+ 61us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:25.772792 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:25.772794 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:25.772857 (+ 63us) tablet.cc:699] Row locks acquired
0804 14:52:25.772860 (+ 3us) write_op.cc:260] PREPARE: finished
0804 14:52:25.772899 (+ 39us) write_op.cc:270] Start()
0804 14:52:26.241086 (+468187us) write_op.cc:276] Timestamp: P: 1785855145772857 usec, L: 0
0804 14:52:26.241091 (+ 5us) op_driver.cc:331] REPLICATION: starting
0804 14:52:26.241328 (+ 237us) log.cc:842] Serialized 354 byte log entry
0804 14:52:27.163762 (+922434us) op_driver.cc:444] REPLICATION: finished
0804 14:52:27.164085 (+ 323us) write_op.cc:301] APPLY: starting
0804 14:52:27.164107 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:27.164138 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:27.164158 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:27.164229 (+ 71us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:27.164256 (+ 27us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:27.164259 (+ 3us) write_op.cc:312] APPLY: finished
0804 14:52:27.164335 (+ 76us) log.cc:842] Serialized 22 byte log entry
0804 14:52:27.164505 (+ 170us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:27.164522 (+ 17us) write_op.cc:454] Released schema lock
0804 14:52:27.164531 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":234,"num_ops":1,"prepare.queue_time_us":68,"prepare.run_cpu_time_us":688,"prepare.run_wall_time_us":468761,"replication_time_us":922626,"thread_start_us":110,"threads_started":1}]]}
I20260804 14:52:27.522502 28715 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:28.068300 28714 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:28.075518 28767 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=44, attempt_no=0}) took 1459 ms. Trace:
I20260804 14:52:28.075659 28767 rpcz_store.cc:276] 0804 14:52:26.616274 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:26.616349 (+ 75us) service_pool.cc:227] Handling call
0804 14:52:28.075494 (+1459145us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:26.616810 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:26.616895 (+ 85us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:26.616898 (+ 3us) write_op.cc:435] Acquired schema lock
0804 14:52:26.616899 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:26.617042 (+ 143us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:26.617051 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:26.617054 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:26.617103 (+ 49us) tablet.cc:699] Row locks acquired
0804 14:52:26.617104 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:26.617143 (+ 39us) write_op.cc:270] Start()
0804 14:52:27.036342 (+419199us) write_op.cc:276] Timestamp: P: 1785855146617107 usec, L: 0
0804 14:52:27.036348 (+ 6us) op_driver.cc:331] REPLICATION: starting
0804 14:52:27.036634 (+ 286us) log.cc:842] Serialized 354 byte log entry
0804 14:52:28.074622 (+1037988us) op_driver.cc:444] REPLICATION: finished
0804 14:52:28.074907 (+ 285us) write_op.cc:301] APPLY: starting
0804 14:52:28.074925 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:28.074958 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:28.074972 (+ 14us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:28.075104 (+ 132us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:28.075144 (+ 40us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:28.075148 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:52:28.075263 (+ 115us) log.cc:842] Serialized 22 byte log entry
0804 14:52:28.075383 (+ 120us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:28.075400 (+ 17us) write_op.cc:454] Released schema lock
0804 14:52:28.075406 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":229,"num_ops":1,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":853,"prepare.run_wall_time_us":419933,"replication_time_us":1038242,"thread_start_us":136,"threads_started":1}]]}
W20260804 14:52:28.193038 28283 tablet_service.cc:2970] error setting up scanner 5be4694d1a6045d181614e5573750f0e with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1785855147192771 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1785855146617107 usec, L: 0 Physical time difference: 0.576s: new_scan_request { tablet_id: "6cd92ff096234dafa214acc72d617f73" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\273\373\307\323\006\"3\n\005slave\022*\n 41255276b38541d68bbfe0d0057a0118\020\001\030\001 \001(\001" signature: "<redacted>"")\215P\256\002\302\361\025)\236\rI]\255\241^\250\030+\004\3636!j\276lc$\025\300l\027N\275O\035T\274\302\3555\200\216\322\212\242\210\302[\342r\211s\325PR\200\310M\227\201x\204\354" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "f38c4acc68bb473786d01674d0ae9341"
I20260804 14:52:28.193393 28283 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:43512 (request call id 28) took 1001 ms. Trace:
I20260804 14:52:28.193442 28283 rpcz_store.cc:276] 0804 14:52:27.192276 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:27.192363 (+ 87us) service_pool.cc:227] Handling call
0804 14:52:27.192575 (+ 212us) tablet_service.cc:2780] Created scanner 5be4694d1a6045d181614e5573750f0e for tablet 6cd92ff096234dafa214acc72d617f73, query id is f38c4acc68bb473786d01674d0ae9341
0804 14:52:27.192786 (+ 211us) tablet_service.cc:2920] Creating iterator
0804 14:52:27.192812 (+ 26us) tablet_service.cc:3297] Waiting safe time to advance
0804 14:52:28.193011 (+1000199us) tablet_service.cc:2945] Iterator created
0804 14:52:28.193027 (+ 16us) tablet_service.cc:2967] Iterator init: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1785855147192771 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1785855146617107 usec, L: 0 Physical time difference: 0.576s
0804 14:52:28.193382 (+ 355us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260804 14:52:28.210561 28715 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:28.210736 28715 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:28.211429 28715 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:52:28.211921 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:28.212004 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:52:28.212589 27891 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:28.393249 28714 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:28.393435 28714 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:28.393985 28714 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:28.394587 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:28.394588 27989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:28.395049 28253 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:28.395264 28714 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:28.405687 28767 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=45, attempt_no=0}) took 1239 ms. Trace:
I20260804 14:52:28.405820 28767 rpcz_store.cc:276] 0804 14:52:27.166298 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:27.166365 (+ 67us) service_pool.cc:227] Handling call
0804 14:52:28.405663 (+1239298us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:27.166857 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:27.166941 (+ 84us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:27.166944 (+ 3us) write_op.cc:435] Acquired schema lock
0804 14:52:27.166946 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:27.167051 (+ 105us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:27.167058 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:27.167059 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:27.167091 (+ 32us) tablet.cc:699] Row locks acquired
0804 14:52:27.167092 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:27.167122 (+ 30us) write_op.cc:270] Start()
0804 14:52:28.077292 (+910170us) write_op.cc:276] Timestamp: P: 1785855147167101 usec, L: 0
0804 14:52:28.077296 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:52:28.077440 (+ 144us) log.cc:842] Serialized 354 byte log entry
0804 14:52:28.405015 (+327575us) op_driver.cc:444] REPLICATION: finished
0804 14:52:28.405169 (+ 154us) write_op.cc:301] APPLY: starting
0804 14:52:28.405195 (+ 26us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:28.405228 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:28.405248 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:28.405316 (+ 68us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:28.405344 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:28.405348 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:52:28.405454 (+ 106us) log.cc:842] Serialized 22 byte log entry
0804 14:52:28.405547 (+ 93us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:28.405563 (+ 16us) write_op.cc:454] Released schema lock
0804 14:52:28.405568 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":67,"num_ops":1,"prepare.queue_time_us":50,"prepare.run_cpu_time_us":583,"prepare.run_wall_time_us":910670,"raft.queue_time_us":159,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":163,"replication_time_us":327675}]]}
I20260804 14:52:28.810446 28714 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:28.810635 28714 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:28.811261 28714 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:28.811666 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:28.811666 27989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:28.812109 28253 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:28.812347 28714 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:29.917686 28714 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:29.917855 28714 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:29.918532 28714 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:29.919055 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 49 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:29.919070 27989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 49 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:29.919607 28252 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:29.919960 28714 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:29.924376 28779 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=47, attempt_no=0}) took 1517 ms. Trace:
I20260804 14:52:29.924510 28779 rpcz_store.cc:276] 0804 14:52:28.407378 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:28.407466 (+ 88us) service_pool.cc:227] Handling call
0804 14:52:29.924356 (+1516890us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:28.408002 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:28.408090 (+ 88us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:28.408093 (+ 3us) write_op.cc:435] Acquired schema lock
0804 14:52:28.408094 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:28.408159 (+ 65us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:28.408165 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:28.408167 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:28.408198 (+ 31us) tablet.cc:699] Row locks acquired
0804 14:52:28.408199 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:28.408228 (+ 29us) write_op.cc:270] Start()
0804 14:52:29.147384 (+739156us) write_op.cc:276] Timestamp: P: 1785855148408154 usec, L: 0
0804 14:52:29.147388 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:52:29.147647 (+ 259us) log.cc:842] Serialized 354 byte log entry
0804 14:52:29.923622 (+775975us) op_driver.cc:444] REPLICATION: finished
0804 14:52:29.923891 (+ 269us) write_op.cc:301] APPLY: starting
0804 14:52:29.923910 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:29.923953 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:29.923968 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:29.924027 (+ 59us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:29.924090 (+ 63us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:29.924093 (+ 3us) write_op.cc:312] APPLY: finished
0804 14:52:29.924159 (+ 66us) log.cc:842] Serialized 22 byte log entry
0804 14:52:29.924249 (+ 90us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:29.924265 (+ 16us) write_op.cc:454] Released schema lock
0804 14:52:29.924271 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":206,"num_ops":1,"prepare.queue_time_us":111,"prepare.run_cpu_time_us":693,"prepare.run_wall_time_us":739756,"raft.queue_time_us":61,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":178,"replication_time_us":776206,"spinlock_wait_cycles":28032,"thread_start_us":88,"threads_started":1}]]}
I20260804 14:52:30.061410 28715 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:30.061823 27988 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.24.6.130:53965 (request call id 3761) took 1849 ms. Trace:
I20260804 14:52:30.061900 27988 rpcz_store.cc:276] 0804 14:52:28.212093 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:28.212177 (+ 84us) service_pool.cc:227] Handling call
0804 14:52:28.212347 (+ 170us) raft_consensus.cc:1457] Updating replica for 2 ops
0804 14:52:28.212466 (+ 119us) raft_consensus.cc:1520] Early marking committed up to index 46
0804 14:52:28.212686 (+ 220us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0804 14:52:28.212814 (+ 128us) write_op.cc:270] Start()
0804 14:52:29.073993 (+861179us) write_op.cc:276] Timestamp: P: 1785855147167101 usec, L: 0
0804 14:52:29.074380 (+ 387us) write_op.cc:270] Start()
0804 14:52:30.060611 (+986231us) write_op.cc:276] Timestamp: P: 1785855148077998 usec, L: 0
0804 14:52:30.061148 (+ 537us) log.cc:842] Serialized 708 byte log entry
0804 14:52:30.061299 (+ 151us) raft_consensus.cc:1631] Marking committed up to 46
0804 14:52:30.061306 (+ 7us) raft_consensus.cc:1683] Filling consensus response to leader.
0804 14:52:30.061340 (+ 34us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0804 14:52:30.061751 (+ 411us) raft_consensus.cc:1671] finished
0804 14:52:30.061753 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0804 14:52:30.061806 (+ 53us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:29.074335 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:29.074450 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:29.074453 (+ 3us) write_op.cc:435] Acquired schema lock
0804 14:52:29.074455 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:29.074524 (+ 69us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:29.074534 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:29.074536 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:29.074573 (+ 37us) tablet.cc:699] Row locks acquired
0804 14:52:29.074574 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0804 14:52:30.061054 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:30.061162 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:30.061166 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:52:30.061167 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:30.061239 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:30.061249 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:30.061251 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:30.061289 (+ 38us) tablet.cc:699] Row locks acquired
0804 14:52:30.061290 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":97,"threads_started":1,"wal-append.queue_time_us":165,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":213,"prepare.run_cpu_time_us":302,"prepare.run_wall_time_us":301,"thread_start_us":118,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":258,"prepare.run_cpu_time_us":342,"prepare.run_wall_time_us":342,"thread_start_us":101,"threads_started":1}]]}
W20260804 14:52:30.140187 28283 tablet_service.cc:2970] error setting up scanner 8f17925604bf4371ad6339a76812e0a1 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.220s (maximum allowed 0.100s): new_scan_request { tablet_id: "6cd92ff096234dafa214acc72d617f73" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\273\373\307\323\006\"3\n\005slave\022*\n 41255276b38541d68bbfe0d0057a0118\020\001\030\001 \001(\001" signature: "<redacted>"")\215P\256\002\302\361\025)\236\rI]\255\241^\250\030+\004\3636!j\276lc$\025\300l\027N\275O\035T\274\302\3555\200\216\322\212\242\210\302[\342r\211s\325PR\200\310M\227\201x\204\354" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "a778d080873b4800b57ddf4a1fab5db6"
I20260804 14:52:30.465240 28714 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:30.465443 28714 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:30.466043 28714 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:30.466495 27989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:30.466708 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:30.467154 28252 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:30.467448 28714 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:30.471547 28789 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39756 (ReqId={client: f7f217c1ca6044288c0f8cb1ca486b56, seq_no=48, attempt_no=0}) took 1627 ms. Trace:
I20260804 14:52:30.471668 28789 rpcz_store.cc:276] 0804 14:52:28.844102 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:28.844181 (+ 79us) service_pool.cc:227] Handling call
0804 14:52:30.471435 (+1627254us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:29.147788 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:29.147906 (+ 118us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:29.147910 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:52:29.147912 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:29.148017 (+ 105us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:29.148027 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:29.148030 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:29.148081 (+ 51us) tablet.cc:699] Row locks acquired
0804 14:52:29.148084 (+ 3us) write_op.cc:260] PREPARE: finished
0804 14:52:29.148136 (+ 52us) write_op.cc:270] Start()
0804 14:52:30.022288 (+874152us) write_op.cc:276] Timestamp: P: 1785855149148092 usec, L: 0
0804 14:52:30.022292 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:52:30.022467 (+ 175us) log.cc:842] Serialized 354 byte log entry
0804 14:52:30.470654 (+448187us) op_driver.cc:444] REPLICATION: finished
0804 14:52:30.470931 (+ 277us) write_op.cc:301] APPLY: starting
0804 14:52:30.470948 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:30.470987 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:30.471008 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:30.471065 (+ 57us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:30.471089 (+ 24us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:30.471093 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:52:30.471163 (+ 70us) log.cc:842] Serialized 22 byte log entry
0804 14:52:30.471297 (+ 134us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:30.471345 (+ 48us) write_op.cc:454] Released schema lock
0804 14:52:30.471352 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":217,"num_ops":1,"prepare.queue_time_us":303128,"prepare.run_cpu_time_us":817,"prepare.run_wall_time_us":874887,"replication_time_us":448334,"spinlock_wait_cycles":768,"thread_start_us":91,"threads_started":1}]]}
W20260804 14:52:31.018517 28466 scanner-internal.cc:458] Time spent opening tablet: real 31.841s user 0.009s sys 0.000s
I20260804 14:52:31.873413 28715 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:31.873582 28715 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:31.874199 28715 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:52:31.874755 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:52:31.874809 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:31.875236 27988 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.24.6.130:53965 (request call id 3762) took 1812 ms. Trace:
I20260804 14:52:31.875608 27893 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:31.875303 27988 rpcz_store.cc:276] 0804 14:52:30.063272 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:30.063337 (+ 65us) service_pool.cc:227] Handling call
0804 14:52:30.063491 (+ 154us) raft_consensus.cc:1457] Updating replica for 2 ops
0804 14:52:30.063603 (+ 112us) raft_consensus.cc:1520] Early marking committed up to index 48
0804 14:52:30.063836 (+ 233us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0804 14:52:30.063911 (+ 75us) write_op.cc:270] Start()
0804 14:52:30.925075 (+861164us) write_op.cc:276] Timestamp: P: 1785855148408154 usec, L: 0
0804 14:52:30.925430 (+ 355us) write_op.cc:270] Start()
0804 14:52:31.872616 (+947186us) write_op.cc:276] Timestamp: P: 1785855149148092 usec, L: 0
0804 14:52:31.873099 (+ 483us) log.cc:842] Serialized 708 byte log entry
0804 14:52:31.873217 (+ 118us) raft_consensus.cc:1631] Marking committed up to 49
0804 14:52:31.873357 (+ 140us) raft_consensus.cc:1683] Filling consensus response to leader.
0804 14:52:31.873383 (+ 26us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0804 14:52:31.875159 (+ 1776us) raft_consensus.cc:1671] finished
0804 14:52:31.875160 (+ 1us) raft_consensus.cc:1677] UpdateReplicas() finished
0804 14:52:31.875220 (+ 60us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:30.925388 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:30.925506 (+ 118us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:30.925515 (+ 9us) write_op.cc:435] Acquired schema lock
0804 14:52:30.925517 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:30.925613 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:30.925626 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:30.925629 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:30.925675 (+ 46us) tablet.cc:699] Row locks acquired
0804 14:52:30.925677 (+ 2us) write_op.cc:260] PREPARE: finished
0804 14:52:31.873246 (+947569us) op_driver.cc:444] REPLICATION: finished
0804 14:52:31.873517 (+ 271us) write_op.cc:301] APPLY: starting
0804 14:52:31.873535 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:31.873717 (+ 182us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:31.873719 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:31.874018 (+ 299us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:31.874054 (+ 36us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:31.874075 (+ 21us) write_op.cc:312] APPLY: finished
0804 14:52:31.874142 (+ 67us) log.cc:842] Serialized 22 byte log entry
0804 14:52:31.874184 (+ 42us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:31.874196 (+ 12us) write_op.cc:454] Released schema lock
0804 14:52:31.874203 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0804 14:52:31.874269 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:31.874271 (+ 2us) write_op.cc:454] Released schema lock
Related trace 'op':
0804 14:52:31.872993 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6cd92ff096234dafa214acc72d617f73
0804 14:52:31.873105 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:31.873109 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:52:31.873111 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:31.873199 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:31.873212 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:31.873214 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:31.873260 (+ 46us) tablet.cc:699] Row locks acquired
0804 14:52:31.873261 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":64,"threads_started":1,"wal-append.queue_time_us":134,"child_traces":[["op",{"apply.queue_time_us":223,"apply.run_cpu_time_us":423,"apply.run_wall_time_us":792,"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":199,"prepare.run_cpu_time_us":363,"prepare.run_wall_time_us":362,"replication_time_us":1809323,"thread_start_us":178,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":220,"prepare.run_cpu_time_us":401,"prepare.run_wall_time_us":399,"thread_start_us":119,"threads_started":1}]]}
I20260804 14:52:31.875774 28715 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:32.312486 28796 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:32.312930 28796 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:32.313853 28796 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:32.313831 27988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:32.314680 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:32.315141 28252 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:32.315546 28796 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:32.993072 28799 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:32.993243 28799 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:32.993717 28800 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:32.993873 28799 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:32.993888 28800 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:32.994203 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:32.994424 27989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:32.995079 28800 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373), 154119e307724bc095278d7e46a3eb05 (127.24.6.131:42591)
I20260804 14:52:32.995414 28252 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:32.995426 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:32.995530 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "785631f53b2c4d8eafac62a9d2719f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "154119e307724bc095278d7e46a3eb05" is_pre_election: true
I20260804 14:52:32.995682 28799 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:32.995985 27893 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 785631f53b2c4d8eafac62a9d2719f58; no voters: 154119e307724bc095278d7e46a3eb05, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:32.996170 28800 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 785631f53b2c4d8eafac62a9d2719f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260804 14:52:34.754004 28467 scanner-internal.cc:458] Time spent opening tablet: real 35.577s user 0.003s sys 0.007s
I20260804 14:52:34.831260 28316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260804 14:52:34.831976 27955 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260804 14:52:34.832984 28135 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260804 14:52:34.844652 28835 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:34.847301 28835 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:34.848022 28835 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:34.848207 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:34.848533 27988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:34.848942 28253 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:34.849442 28835 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
0a55c607a4694a67bf71a63db589c9d7 | 127.24.6.190:37795 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.24.6.148:42301 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
154119e307724bc095278d7e46a3eb05 | 127.24.6.131:42591 | HEALTHY | <none> | 0 | 0
785631f53b2c4d8eafac62a9d2719f58 | 127.24.6.129:43707 | HEALTHY | <none> | 0 | 0
92ba21101c0a4918be8e047240f43c8b | 127.24.6.130:43373 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
consensus_inject_latency_ms_in_notifications | 50 | unsafe,hidden | all 3 server(s) checked
inject_latency_ms_before_starting_op | 1000 | hidden,advanced | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.24.6.129 | experimental | 127.24.6.129:43707
local_ip_for_outbound_sockets | 127.24.6.130 | experimental | 127.24.6.130:43373
local_ip_for_outbound_sockets | 127.24.6.131 | experimental | 127.24.6.131:42591
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.24.6.129:43707
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.24.6.130:43373
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.24.6.131:42591
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.24.6.148:42301 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260804 14:52:34.864796 24602 log_verifier.cc:126] Checking tablet 6cd92ff096234dafa214acc72d617f73
I20260804 14:52:34.899400 28835 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:34.899559 28835 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:34.900161 28835 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:34.900656 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
I20260804 14:52:34.903522 27988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "785631f53b2c4d8eafac62a9d2719f58" is_pre_election: true
I20260804 14:52:34.904192 28253 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:34.904538 28835 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:34.911063 24602 log_verifier.cc:177] Verified matching terms for 50 ops in tablet 6cd92ff096234dafa214acc72d617f73
I20260804 14:52:34.935716 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 27869
I20260804 14:52:34.941058 28835 raft_consensus.cc:482] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92ba21101c0a4918be8e047240f43c8b)
I20260804 14:52:34.941229 28835 raft_consensus.cc:504] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785631f53b2c4d8eafac62a9d2719f58" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43707 } } peers { permanent_uuid: "92ba21101c0a4918be8e047240f43c8b" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 43373 } } peers { permanent_uuid: "154119e307724bc095278d7e46a3eb05" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42591 } }
I20260804 14:52:34.941779 28835 leader_election.cc:288] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707), 92ba21101c0a4918be8e047240f43c8b (127.24.6.130:43373)
I20260804 14:52:34.943631 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd92ff096234dafa214acc72d617f73" candidate_uuid: "154119e307724bc095278d7e46a3eb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "92ba21101c0a4918be8e047240f43c8b" is_pre_election: true
W20260804 14:52:34.950994 28253 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.24.6.129:43707 timed out after 0.008s (SENT) [suppressed 1 similar messages]
W20260804 14:52:34.952872 28253 connection.cc:535] client connection to 127.24.6.129:43707 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260804 14:52:34.952890 28077 connection.cc:535] client connection to 127.24.6.129:43707 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260804 14:52:34.953017 28077 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 1 similar messages]
W20260804 14:52:34.953078 28253 leader_election.cc:334] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 785631f53b2c4d8eafac62a9d2719f58 (127.24.6.129:43707): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260804 14:52:34.953156 28253 leader_election.cc:302] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 154119e307724bc095278d7e46a3eb05; no voters: 785631f53b2c4d8eafac62a9d2719f58, 92ba21101c0a4918be8e047240f43c8b
I20260804 14:52:34.953229 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 28055
I20260804 14:52:34.959662 28835 raft_consensus.cc:2726] T 6cd92ff096234dafa214acc72d617f73 P 154119e307724bc095278d7e46a3eb05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:34.966233 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 28234
I20260804 14:52:34.981729 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 27735
2026-08-04T14:52:34Z chronyd exiting
[ OK ] RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency (37065 ms)
[ RUN ] RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys
I20260804 14:52:35.013518 24602 test_util.cc:268] Using random seed: -836597880
I20260804 14:52:35.014413 24602 ts_itest-base.cc:112] Starting cluster with:
I20260804 14:52:35.014485 24602 ts_itest-base.cc:113] --------------
I20260804 14:52:35.014506 24602 ts_itest-base.cc:114] 3 tablet servers
I20260804 14:52:35.014531 24602 ts_itest-base.cc:115] 3 replicas per TS
I20260804 14:52:35.014555 24602 ts_itest-base.cc:116] --------------
2026-08-04T14:52:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-04T14:52:35Z Disabled control of system clock
I20260804 14:52:35.025976 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.6.190:39627
--webserver_interface=127.24.6.190
--webserver_port=0
--builtin_ntp_servers=127.24.6.148:34763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.6.190:39627 with env {}
W20260804 14:52:35.146375 28868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:52:35.146765 28868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:52:35.146847 28868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:52:35.151027 28868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260804 14:52:35.151130 28868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:52:35.151162 28868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260804 14:52:35.151243 28868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260804 14:52:35.156519 28868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:34763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.6.190:39627
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.6.190:39627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.24.6.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.28868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:52:35.157740 28868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:52:35.159269 28868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:52:35.168772 28873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:52:35.168726 28876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:52:35.168836 28868 server_base.cc:1034] running on GCE node
W20260804 14:52:35.168749 28874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:52:35.169848 28868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:52:35.171054 28868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:52:35.172214 28868 hybrid_clock.cc:648] HybridClock initialized: now 1785855155172173 us; error 59 us; skew 500 ppm
I20260804 14:52:35.174240 28868 webserver.cc:460] Webserver started at http://127.24.6.190:46219/ using document root <none> and password file <none>
I20260804 14:52:35.174880 28868 fs_manager.cc:359] Metadata directory not provided
I20260804 14:52:35.174944 28868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:52:35.175190 28868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:52:35.177423 28868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d48d86cb47d943c8b62c82944326ff2d"
format_stamp: "Formatted at 2026-08-04 14:52:35 on dist-test-slave-92m1"
I20260804 14:52:35.178035 28868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d48d86cb47d943c8b62c82944326ff2d"
format_stamp: "Formatted at 2026-08-04 14:52:35 on dist-test-slave-92m1"
I20260804 14:52:35.182363 28868 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260804 14:52:35.185191 28882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:52:35.186627 28868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260804 14:52:35.186825 28868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d48d86cb47d943c8b62c82944326ff2d"
format_stamp: "Formatted at 2026-08-04 14:52:35 on dist-test-slave-92m1"
I20260804 14:52:35.186966 28868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:52:35.206604 28868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:52:35.207499 28868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:52:35.207731 28868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:52:35.215713 28868 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.190:39627
I20260804 14:52:35.215736 28952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.190:39627 every 8 connection(s)
I20260804 14:52:35.216758 28868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260804 14:52:35.220633 28953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:52:35.223011 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 28868
I20260804 14:52:35.223145 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260804 14:52:35.226884 28953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d48d86cb47d943c8b62c82944326ff2d: Bootstrap starting.
I20260804 14:52:35.229499 28953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d48d86cb47d943c8b62c82944326ff2d: Neither blocks nor log segments found. Creating new log.
I20260804 14:52:35.230526 28953 log.cc:824] T 00000000000000000000000000000000 P d48d86cb47d943c8b62c82944326ff2d: Log is configured to *not* fsync() on all Append() calls
I20260804 14:52:35.232726 28953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d48d86cb47d943c8b62c82944326ff2d: No bootstrap required, opened a new log
I20260804 14:52:35.237527 28953 raft_consensus.cc:348] T 00000000000000000000000000000000 P d48d86cb47d943c8b62c82944326ff2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d48d86cb47d943c8b62c82944326ff2d" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 39627 } }
I20260804 14:52:35.237773 28953 raft_consensus.cc:374] T 00000000000000000000000000000000 P d48d86cb47d943c8b62c82944326ff2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:52:35.237865 28953 raft_consensus.cc:729] T 00000000000000000000000000000000 P d48d86cb47d943c8b62c82944326ff2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d48d86cb47d943c8b62c82944326ff2d, State: Initialized, Role: FOLLOWER
I20260804 14:52:35.238332 28953 consensus_queue.cc:260] T 00000000000000000000000000000000 P d48d86cb47d943c8b62c82944326ff2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d48d86cb47d943c8b62c82944326ff2d" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 39627 } }
I20260804 14:52:35.238495 28953 raft_consensus.cc:388] T 00000000000000000000000000000000 P d48d86cb47d943c8b62c82944326ff2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260804 14:52:35.238564 28953 raft_consensus.cc:482] T 00000000000000000000000000000000 P d48d86cb47d943c8b62c82944326ff2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260804 14:52:35.238703 28953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d48d86cb47d943c8b62c82944326ff2d [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:52:35.239743 28953 raft_consensus.cc:504] T 00000000000000000000000000000000 P d48d86cb47d943c8b62c82944326ff2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d48d86cb47d943c8b62c82944326ff2d" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 39627 } }
I20260804 14:52:35.240126 28953 leader_election.cc:302] T 00000000000000000000000000000000 P d48d86cb47d943c8b62c82944326ff2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d48d86cb47d943c8b62c82944326ff2d; no voters:
I20260804 14:52:35.240476 28953 leader_election.cc:288] T 00000000000000000000000000000000 P d48d86cb47d943c8b62c82944326ff2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260804 14:52:35.240622 28958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d48d86cb47d943c8b62c82944326ff2d [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:52:35.240860 28958 raft_consensus.cc:686] T 00000000000000000000000000000000 P d48d86cb47d943c8b62c82944326ff2d [term 1 LEADER]: Becoming Leader. State: Replica: d48d86cb47d943c8b62c82944326ff2d, State: Running, Role: LEADER
I20260804 14:52:35.241271 28958 consensus_queue.cc:237] T 00000000000000000000000000000000 P d48d86cb47d943c8b62c82944326ff2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d48d86cb47d943c8b62c82944326ff2d" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 39627 } }
I20260804 14:52:35.242018 28953 sys_catalog.cc:564] T 00000000000000000000000000000000 P d48d86cb47d943c8b62c82944326ff2d [sys.catalog]: configured and running, proceeding with master startup.
I20260804 14:52:35.243268 28959 sys_catalog.cc:455] T 00000000000000000000000000000000 P d48d86cb47d943c8b62c82944326ff2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d48d86cb47d943c8b62c82944326ff2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d48d86cb47d943c8b62c82944326ff2d" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 39627 } } }
I20260804 14:52:35.243410 28959 sys_catalog.cc:458] T 00000000000000000000000000000000 P d48d86cb47d943c8b62c82944326ff2d [sys.catalog]: This master's current role is: LEADER
I20260804 14:52:35.243244 28960 sys_catalog.cc:455] T 00000000000000000000000000000000 P d48d86cb47d943c8b62c82944326ff2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d48d86cb47d943c8b62c82944326ff2d. Latest consensus state: current_term: 1 leader_uuid: "d48d86cb47d943c8b62c82944326ff2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d48d86cb47d943c8b62c82944326ff2d" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 39627 } } }
I20260804 14:52:35.243705 28960 sys_catalog.cc:458] T 00000000000000000000000000000000 P d48d86cb47d943c8b62c82944326ff2d [sys.catalog]: This master's current role is: LEADER
I20260804 14:52:35.243842 28970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260804 14:52:35.246974 28970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260804 14:52:35.253741 28970 catalog_manager.cc:1347] Generated new cluster ID: 602cab049bb149aeafd12666f567e1b8
I20260804 14:52:35.253829 28970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260804 14:52:35.260421 28970 catalog_manager.cc:1370] Generated new certificate authority record
I20260804 14:52:35.261404 28970 catalog_manager.cc:1504] Loading token signing keys...
I20260804 14:52:35.268878 28970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d48d86cb47d943c8b62c82944326ff2d: Generated new TSK 0
I20260804 14:52:35.269706 28970 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260804 14:52:35.280989 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.129:0
--local_ip_for_outbound_sockets=127.24.6.129
--webserver_interface=127.24.6.129
--webserver_port=0
--tserver_master_addrs=127.24.6.190:39627
--builtin_ntp_servers=127.24.6.148:34763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
W20260804 14:52:35.404119 28982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:52:35.404431 28982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:52:35.404472 28982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:52:35.408840 28982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:52:35.408964 28982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.129
I20260804 14:52:35.414089 28982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:34763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.24.6.129
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.24.6.190:39627
--never_fsync=true
--heap_profile_path=/tmp/kudu.28982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.129
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:52:35.415371 28982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:52:35.416916 28982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:52:35.427685 28996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:52:35.427696 28991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:52:35.427716 28990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:52:35.428071 28982 server_base.cc:1034] running on GCE node
I20260804 14:52:35.428610 28982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:52:35.429360 28982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:52:35.430578 28982 hybrid_clock.cc:648] HybridClock initialized: now 1785855155430560 us; error 31 us; skew 500 ppm
I20260804 14:52:35.432830 28982 webserver.cc:460] Webserver started at http://127.24.6.129:33785/ using document root <none> and password file <none>
I20260804 14:52:35.433535 28982 fs_manager.cc:359] Metadata directory not provided
I20260804 14:52:35.433643 28982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:52:35.433899 28982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:52:35.436095 28982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "02bbbb8aa9ba40d492afb93419678550"
format_stamp: "Formatted at 2026-08-04 14:52:35 on dist-test-slave-92m1"
I20260804 14:52:35.436723 28982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "02bbbb8aa9ba40d492afb93419678550"
format_stamp: "Formatted at 2026-08-04 14:52:35 on dist-test-slave-92m1"
I20260804 14:52:35.441578 28982 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260804 14:52:35.444759 29002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:52:35.446259 28982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260804 14:52:35.446446 28982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "02bbbb8aa9ba40d492afb93419678550"
format_stamp: "Formatted at 2026-08-04 14:52:35 on dist-test-slave-92m1"
I20260804 14:52:35.446591 28982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:52:35.474116 28982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:52:35.474912 28982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:52:35.475143 28982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:52:35.475811 28982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:52:35.476868 28982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:52:35.476943 28982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:52:35.477028 28982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:52:35.477073 28982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:52:35.486769 28982 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.129:41707
I20260804 14:52:35.486796 29157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.129:41707 every 8 connection(s)
I20260804 14:52:35.487854 28982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260804 14:52:35.489732 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 28982
I20260804 14:52:35.489882 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260804 14:52:35.492673 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.130:0
--local_ip_for_outbound_sockets=127.24.6.130
--webserver_interface=127.24.6.130
--webserver_port=0
--tserver_master_addrs=127.24.6.190:39627
--builtin_ntp_servers=127.24.6.148:34763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
I20260804 14:52:35.500286 29158 heartbeater.cc:347] Connected to a master server at 127.24.6.190:39627
I20260804 14:52:35.500612 29158 heartbeater.cc:464] Registering TS with master...
I20260804 14:52:35.501219 29158 heartbeater.cc:511] Master 127.24.6.190:39627 requested a full tablet report, sending...
I20260804 14:52:35.503021 28902 ts_manager.cc:194] Registered new tserver with Master: 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:52:35.504297 28902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.129:43505
W20260804 14:52:35.625249 29162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:52:35.625627 29162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:52:35.625717 29162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:52:35.630182 29162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:52:35.630380 29162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.130
I20260804 14:52:35.635674 29162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:34763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.24.6.130
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.24.6.190:39627
--never_fsync=true
--heap_profile_path=/tmp/kudu.29162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.130
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:52:35.636943 29162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:52:35.638468 29162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:52:35.649245 29170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:52:35.649248 29171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:52:35.649268 29173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:52:35.649566 29162 server_base.cc:1034] running on GCE node
I20260804 14:52:35.650183 29162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:52:35.650947 29162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:52:35.652168 29162 hybrid_clock.cc:648] HybridClock initialized: now 1785855155652116 us; error 62 us; skew 500 ppm
I20260804 14:52:35.654347 29162 webserver.cc:460] Webserver started at http://127.24.6.130:34975/ using document root <none> and password file <none>
I20260804 14:52:35.655018 29162 fs_manager.cc:359] Metadata directory not provided
I20260804 14:52:35.655120 29162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:52:35.655413 29162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:52:35.657548 29162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "a310b164338b437bb1577d62a34897a0"
format_stamp: "Formatted at 2026-08-04 14:52:35 on dist-test-slave-92m1"
I20260804 14:52:35.658162 29162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "a310b164338b437bb1577d62a34897a0"
format_stamp: "Formatted at 2026-08-04 14:52:35 on dist-test-slave-92m1"
I20260804 14:52:35.662654 29162 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260804 14:52:35.665467 29184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:52:35.667181 29162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260804 14:52:35.667440 29162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "a310b164338b437bb1577d62a34897a0"
format_stamp: "Formatted at 2026-08-04 14:52:35 on dist-test-slave-92m1"
I20260804 14:52:35.667583 29162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:52:35.682375 29162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:52:35.683194 29162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:52:35.683473 29162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:52:35.684186 29162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:52:35.685272 29162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:52:35.685349 29162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:52:35.685421 29162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:52:35.685452 29162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:52:35.695657 29162 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.130:33743
I20260804 14:52:35.695688 29334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.130:33743 every 8 connection(s)
I20260804 14:52:35.696813 29162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260804 14:52:35.703397 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 29162
I20260804 14:52:35.703567 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260804 14:52:35.706349 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.131:0
--local_ip_for_outbound_sockets=127.24.6.131
--webserver_interface=127.24.6.131
--webserver_port=0
--tserver_master_addrs=127.24.6.190:39627
--builtin_ntp_servers=127.24.6.148:34763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
I20260804 14:52:35.709270 29335 heartbeater.cc:347] Connected to a master server at 127.24.6.190:39627
I20260804 14:52:35.709617 29335 heartbeater.cc:464] Registering TS with master...
I20260804 14:52:35.710379 29335 heartbeater.cc:511] Master 127.24.6.190:39627 requested a full tablet report, sending...
I20260804 14:52:35.711686 28902 ts_manager.cc:194] Registered new tserver with Master: a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:52:35.712735 28902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.130:47791
W20260804 14:52:35.833082 29342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:52:35.833408 29342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:52:35.833451 29342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:52:35.837939 29342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:52:35.838127 29342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.131
I20260804 14:52:35.843778 29342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:34763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.24.6.131
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.24.6.190:39627
--never_fsync=true
--heap_profile_path=/tmp/kudu.29342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.131
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:52:35.845161 29342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:52:35.846732 29342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:52:35.857735 29351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:52:35.857776 29349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:52:35.857805 29342 server_base.cc:1034] running on GCE node
W20260804 14:52:35.857735 29353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:52:35.858639 29342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:52:35.859402 29342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:52:35.860600 29342 hybrid_clock.cc:648] HybridClock initialized: now 1785855155860559 us; error 56 us; skew 500 ppm
I20260804 14:52:35.863111 29342 webserver.cc:460] Webserver started at http://127.24.6.131:34057/ using document root <none> and password file <none>
I20260804 14:52:35.863905 29342 fs_manager.cc:359] Metadata directory not provided
I20260804 14:52:35.863994 29342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:52:35.864288 29342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:52:35.866683 29342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "30a15be32d8d43a6a4f432dd9b6a6869"
format_stamp: "Formatted at 2026-08-04 14:52:35 on dist-test-slave-92m1"
I20260804 14:52:35.867457 29342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "30a15be32d8d43a6a4f432dd9b6a6869"
format_stamp: "Formatted at 2026-08-04 14:52:35 on dist-test-slave-92m1"
I20260804 14:52:35.871922 29342 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260804 14:52:35.874959 29361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:52:35.876631 29342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260804 14:52:35.876799 29342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "30a15be32d8d43a6a4f432dd9b6a6869"
format_stamp: "Formatted at 2026-08-04 14:52:35 on dist-test-slave-92m1"
I20260804 14:52:35.876940 29342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:52:35.894526 29342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:52:35.895448 29342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:52:35.895673 29342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:52:35.896344 29342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:52:35.897382 29342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:52:35.897457 29342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:52:35.897536 29342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:52:35.897581 29342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:52:35.907171 29342 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.131:38993
I20260804 14:52:35.907191 29512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.131:38993 every 8 connection(s)
I20260804 14:52:35.908257 29342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260804 14:52:35.916296 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 29342
I20260804 14:52:35.916435 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260804 14:52:35.921041 29513 heartbeater.cc:347] Connected to a master server at 127.24.6.190:39627
I20260804 14:52:35.921319 29513 heartbeater.cc:464] Registering TS with master...
I20260804 14:52:35.921923 29513 heartbeater.cc:511] Master 127.24.6.190:39627 requested a full tablet report, sending...
I20260804 14:52:35.923095 28902 ts_manager.cc:194] Registered new tserver with Master: 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993)
I20260804 14:52:35.923802 28902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.131:40699
I20260804 14:52:35.932235 24602 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260804 14:52:35.934193 24602 test_util.cc:268] Using random seed: -835677205
I20260804 14:52:35.948207 28902 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55152:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260804 14:52:35.949541 28902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260804 14:52:35.968956 29066 tablet_service.cc:1467] Processing CreateTablet for tablet 0118633f5af24844b38a60678228343e (DEFAULT_TABLE table=test-workload [id=17b6e966c8434751b3a9c124c7dbbb2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:52:35.968953 29420 tablet_service.cc:1467] Processing CreateTablet for tablet 0118633f5af24844b38a60678228343e (DEFAULT_TABLE table=test-workload [id=17b6e966c8434751b3a9c124c7dbbb2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:52:35.970046 29420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0118633f5af24844b38a60678228343e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:52:35.970063 29066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0118633f5af24844b38a60678228343e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:52:35.971863 29252 tablet_service.cc:1467] Processing CreateTablet for tablet 0118633f5af24844b38a60678228343e (DEFAULT_TABLE table=test-workload [id=17b6e966c8434751b3a9c124c7dbbb2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:52:35.973096 29252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0118633f5af24844b38a60678228343e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:52:35.976591 29537 tablet_bootstrap.cc:492] T 0118633f5af24844b38a60678228343e P 30a15be32d8d43a6a4f432dd9b6a6869: Bootstrap starting.
I20260804 14:52:35.977283 29538 tablet_bootstrap.cc:492] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550: Bootstrap starting.
I20260804 14:52:35.979559 29537 tablet_bootstrap.cc:654] T 0118633f5af24844b38a60678228343e P 30a15be32d8d43a6a4f432dd9b6a6869: Neither blocks nor log segments found. Creating new log.
I20260804 14:52:35.979559 29538 tablet_bootstrap.cc:654] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550: Neither blocks nor log segments found. Creating new log.
I20260804 14:52:35.980477 29539 tablet_bootstrap.cc:492] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0: Bootstrap starting.
I20260804 14:52:35.980532 29537 log.cc:824] T 0118633f5af24844b38a60678228343e P 30a15be32d8d43a6a4f432dd9b6a6869: Log is configured to *not* fsync() on all Append() calls
I20260804 14:52:35.980545 29538 log.cc:824] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550: Log is configured to *not* fsync() on all Append() calls
I20260804 14:52:35.983042 29537 tablet_bootstrap.cc:492] T 0118633f5af24844b38a60678228343e P 30a15be32d8d43a6a4f432dd9b6a6869: No bootstrap required, opened a new log
I20260804 14:52:35.983186 29538 tablet_bootstrap.cc:492] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550: No bootstrap required, opened a new log
I20260804 14:52:35.983283 29539 tablet_bootstrap.cc:654] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0: Neither blocks nor log segments found. Creating new log.
I20260804 14:52:35.983314 29537 ts_tablet_manager.cc:1397] T 0118633f5af24844b38a60678228343e P 30a15be32d8d43a6a4f432dd9b6a6869: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260804 14:52:35.983433 29538 ts_tablet_manager.cc:1397] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20260804 14:52:35.984186 29539 log.cc:824] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0: Log is configured to *not* fsync() on all Append() calls
I20260804 14:52:35.986617 29539 tablet_bootstrap.cc:492] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0: No bootstrap required, opened a new log
I20260804 14:52:35.986861 29539 ts_tablet_manager.cc:1397] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260804 14:52:35.987304 29537 raft_consensus.cc:348] T 0118633f5af24844b38a60678228343e P 30a15be32d8d43a6a4f432dd9b6a6869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:35.987591 29537 raft_consensus.cc:374] T 0118633f5af24844b38a60678228343e P 30a15be32d8d43a6a4f432dd9b6a6869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:52:35.987666 29537 raft_consensus.cc:729] T 0118633f5af24844b38a60678228343e P 30a15be32d8d43a6a4f432dd9b6a6869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a15be32d8d43a6a4f432dd9b6a6869, State: Initialized, Role: FOLLOWER
I20260804 14:52:35.987933 29538 raft_consensus.cc:348] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:35.988307 29538 raft_consensus.cc:374] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:52:35.988291 29537 consensus_queue.cc:260] T 0118633f5af24844b38a60678228343e P 30a15be32d8d43a6a4f432dd9b6a6869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:35.988415 29538 raft_consensus.cc:729] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02bbbb8aa9ba40d492afb93419678550, State: Initialized, Role: FOLLOWER
I20260804 14:52:35.989183 29538 consensus_queue.cc:260] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:35.989482 29537 ts_tablet_manager.cc:1428] T 0118633f5af24844b38a60678228343e P 30a15be32d8d43a6a4f432dd9b6a6869: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260804 14:52:35.989760 29543 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 30a15be32d8d43a6a4f432dd9b6a6869 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260804 14:52:35.989959 29513 heartbeater.cc:503] Master 127.24.6.190:39627 was elected leader, sending a full tablet report...
I20260804 14:52:35.989984 29543 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 30a15be32d8d43a6a4f432dd9b6a6869 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:35.990473 29545 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260804 14:52:35.990548 29158 heartbeater.cc:503] Master 127.24.6.190:39627 was elected leader, sending a full tablet report...
I20260804 14:52:35.990703 29545 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:35.991823 29545 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:52:35.992769 29538 ts_tablet_manager.cc:1428] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550: Time spent starting tablet: real 0.009s user 0.004s sys 0.002s
I20260804 14:52:35.993314 29543 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 30a15be32d8d43a6a4f432dd9b6a6869 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:52:35.995620 29539 raft_consensus.cc:348] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:35.996006 29539 raft_consensus.cc:374] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:52:35.996104 29539 raft_consensus.cc:729] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a310b164338b437bb1577d62a34897a0, State: Initialized, Role: FOLLOWER
I20260804 14:52:35.996678 29539 consensus_queue.cc:260] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:35.997725 29335 heartbeater.cc:503] Master 127.24.6.190:39627 was elected leader, sending a full tablet report...
I20260804 14:52:35.997864 29539 ts_tablet_manager.cc:1428] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260804 14:52:35.999317 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:35.999392 29553 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260804 14:52:35.999567 29450 raft_consensus.cc:2454] T 0118633f5af24844b38a60678228343e P 30a15be32d8d43a6a4f432dd9b6a6869 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02bbbb8aa9ba40d492afb93419678550 in term 0.
I20260804 14:52:35.999563 29553 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
W20260804 14:52:35.999660 29364 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.24.6.130:33743 timed out after 0.006s (ON_OUTBOUND_QUEUE)
W20260804 14:52:35.999847 29006 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.24.6.131:38993 timed out after 0.007s (SENT)
I20260804 14:52:36.004021 29553 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:52:36.006080 29272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:52:36.006383 29272 raft_consensus.cc:2454] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30a15be32d8d43a6a4f432dd9b6a6869 in term 0.
I20260804 14:52:36.006403 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:36.006532 29450 raft_consensus.cc:2454] T 0118633f5af24844b38a60678228343e P 30a15be32d8d43a6a4f432dd9b6a6869 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02bbbb8aa9ba40d492afb93419678550 in term 0.
I20260804 14:52:36.006934 29364 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 30a15be32d8d43a6a4f432dd9b6a6869 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0; no voters:
I20260804 14:52:36.007082 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869; no voters:
I20260804 14:52:36.007304 29544 raft_consensus.cc:2781] T 0118633f5af24844b38a60678228343e P 30a15be32d8d43a6a4f432dd9b6a6869 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260804 14:52:36.007409 29544 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 30a15be32d8d43a6a4f432dd9b6a6869 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260804 14:52:36.007411 29547 raft_consensus.cc:2781] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260804 14:52:36.007465 29544 raft_consensus.cc:3037] T 0118633f5af24844b38a60678228343e P 30a15be32d8d43a6a4f432dd9b6a6869 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:52:36.007499 29547 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260804 14:52:36.007550 29547 raft_consensus.cc:3037] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 0 FOLLOWER]: Advancing to term 1
W20260804 14:52:36.007555 29187 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.24.6.131:38993 timed out after 0.006s (SENT)
I20260804 14:52:36.008257 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:36.008777 29544 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 30a15be32d8d43a6a4f432dd9b6a6869 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:36.008777 29547 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:36.009027 29272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:52:36.009099 29450 raft_consensus.cc:2379] T 0118633f5af24844b38a60678228343e P 30a15be32d8d43a6a4f432dd9b6a6869 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a310b164338b437bb1577d62a34897a0 in current term 1: Already voted for candidate 30a15be32d8d43a6a4f432dd9b6a6869 in this term.
I20260804 14:52:36.009161 29272 raft_consensus.cc:2454] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02bbbb8aa9ba40d492afb93419678550 in term 0.
I20260804 14:52:36.009184 29547 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 1 election: Requested vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:52:36.009238 29544 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 30a15be32d8d43a6a4f432dd9b6a6869 [CANDIDATE]: Term 1 election: Requested vote from peers a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:52:36.009742 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869"
I20260804 14:52:36.009883 29450 raft_consensus.cc:2379] T 0118633f5af24844b38a60678228343e P 30a15be32d8d43a6a4f432dd9b6a6869 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 02bbbb8aa9ba40d492afb93419678550 in current term 1: Already voted for candidate 30a15be32d8d43a6a4f432dd9b6a6869 in this term.
I20260804 14:52:36.009868 29097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550"
I20260804 14:52:36.010102 29097 raft_consensus.cc:2379] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 30a15be32d8d43a6a4f432dd9b6a6869 in current term 1: Already voted for candidate 02bbbb8aa9ba40d492afb93419678550 in this term.
I20260804 14:52:36.010219 29096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:52:36.010355 29096 raft_consensus.cc:2379] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30a15be32d8d43a6a4f432dd9b6a6869 in current term 1: Already voted for candidate 02bbbb8aa9ba40d492afb93419678550 in this term.
I20260804 14:52:36.010680 29272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0"
I20260804 14:52:36.010695 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0"
I20260804 14:52:36.010834 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:36.010823 29272 raft_consensus.cc:3037] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:52:36.010946 29450 raft_consensus.cc:2379] T 0118633f5af24844b38a60678228343e P 30a15be32d8d43a6a4f432dd9b6a6869 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a310b164338b437bb1577d62a34897a0 in current term 1: Already voted for candidate 30a15be32d8d43a6a4f432dd9b6a6869 in this term.
I20260804 14:52:36.011322 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:52:36.011592 29547 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260804 14:52:36.012216 29272 raft_consensus.cc:2454] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30a15be32d8d43a6a4f432dd9b6a6869 in term 1.
I20260804 14:52:36.012518 29364 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 30a15be32d8d43a6a4f432dd9b6a6869 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550
I20260804 14:52:36.012773 29544 raft_consensus.cc:2781] T 0118633f5af24844b38a60678228343e P 30a15be32d8d43a6a4f432dd9b6a6869 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:52:36.012972 29544 raft_consensus.cc:686] T 0118633f5af24844b38a60678228343e P 30a15be32d8d43a6a4f432dd9b6a6869 [term 1 LEADER]: Becoming Leader. State: Replica: 30a15be32d8d43a6a4f432dd9b6a6869, State: Running, Role: LEADER
I20260804 14:52:36.013377 29096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:52:36.013536 29096 raft_consensus.cc:2379] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a310b164338b437bb1577d62a34897a0 in current term 1: Already voted for candidate 02bbbb8aa9ba40d492afb93419678550 in this term.
I20260804 14:52:36.013474 29544 consensus_queue.cc:237] T 0118633f5af24844b38a60678228343e P 30a15be32d8d43a6a4f432dd9b6a6869 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:36.013808 29185 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:52:36.014093 29553 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260804 14:52:36.017102 29544 consensus_queue.cc:1035] T 0118633f5af24844b38a60678228343e P 30a15be32d8d43a6a4f432dd9b6a6869 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260804 14:52:36.017441 29543 consensus_queue.cc:1035] T 0118633f5af24844b38a60678228343e P 30a15be32d8d43a6a4f432dd9b6a6869 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260804 14:52:36.017952 28902 catalog_manager.cc:5495] T 0118633f5af24844b38a60678228343e P 30a15be32d8d43a6a4f432dd9b6a6869 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131). New cstate: current_term: 1 leader_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } health_report { overall_health: UNKNOWN } } }
W20260804 14:52:36.043277 29561 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260804 14:52:36.043380 29561 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260804 14:52:36.493223 29553 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:36.493373 29553 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:36.493805 29553 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:52:36.494138 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:36.495862 29096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:52:36.496389 29185 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:52:36.496582 29553 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260804 14:52:36.971072 29272 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260804 14:52:36.971261 29272 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20260804 14:52:36.972157 29096 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260804 14:52:36.972226 29096 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260804 14:52:36.977546 29553 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:36.977679 29553 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:36.978111 29553 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:52:36.978847 29097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:52:36.979436 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:36.979956 29187 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:52:36.980232 29553 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:37.729219 29575 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:37.729421 29575 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:37.730369 29575 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:52:37.730880 29272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:52:37.730989 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:37.731501 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:52:37.732376 29575 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:38.370880 29575 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:38.371083 29575 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:38.371752 29575 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:52:38.372365 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:38.372391 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:52:38.372995 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:52:38.373307 29575 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:38.373693 29588 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=0, attempt_no=0}) took 2335 ms. Trace:
I20260804 14:52:38.373816 29588 rpcz_store.cc:276] 0804 14:52:36.038121 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:36.038263 (+ 142us) service_pool.cc:227] Handling call
0804 14:52:38.373672 (+2335409us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:36.969382 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:52:36.969502 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:36.969508 (+ 6us) write_op.cc:435] Acquired schema lock
0804 14:52:36.969511 (+ 3us) tablet.cc:660] Decoding operations
0804 14:52:36.969601 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:36.969640 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:36.969645 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:36.969694 (+ 49us) tablet.cc:699] Row locks acquired
0804 14:52:36.969695 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:36.969743 (+ 48us) write_op.cc:270] Start()
0804 14:52:37.732884 (+763141us) write_op.cc:276] Timestamp: P: 1785855156969719 usec, L: 0
0804 14:52:37.732888 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:52:37.733157 (+ 269us) log.cc:842] Serialized 345 byte log entry
0804 14:52:38.372672 (+639515us) op_driver.cc:444] REPLICATION: finished
0804 14:52:38.372942 (+ 270us) write_op.cc:301] APPLY: starting
0804 14:52:38.372963 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:38.372998 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:38.373015 (+ 17us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:38.373118 (+ 103us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:38.373153 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:38.373158 (+ 5us) write_op.cc:312] APPLY: finished
0804 14:52:38.373236 (+ 78us) log.cc:842] Serialized 22 byte log entry
0804 14:52:38.373563 (+ 327us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:38.373578 (+ 15us) write_op.cc:454] Released schema lock
0804 14:52:38.373586 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":356736,"child_traces":[["op",{"apply.queue_time_us":181,"mutex_wait_us":20,"num_ops":1,"prepare.queue_time_us":928446,"prepare.run_cpu_time_us":830,"prepare.run_wall_time_us":763886,"raft.queue_time_us":330,"raft.run_cpu_time_us":394,"raft.run_wall_time_us":1022,"replication_time_us":639762,"spinlock_wait_cycles":156416,"thread_start_us":68,"threads_started":1}]]}
I20260804 14:52:38.595278 29582 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:38.595477 29582 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:38.596517 29582 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:52:38.597012 29097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:52:38.596984 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:38.597538 29185 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:52:38.909824 29575 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:38.909998 29575 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:38.910573 29575 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:52:38.911108 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:52:38.911602 29595 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=2, attempt_no=0}) took 1927 ms. Trace:
I20260804 14:52:38.911913 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:38.911707 29595 rpcz_store.cc:276] 0804 14:52:36.984176 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:36.984247 (+ 71us) service_pool.cc:227] Handling call
0804 14:52:38.911584 (+1927337us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:37.733315 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:52:37.733424 (+ 109us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:37.733429 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:52:37.733431 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:37.733509 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:37.733519 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:37.733522 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:37.733560 (+ 38us) tablet.cc:699] Row locks acquired
0804 14:52:37.733562 (+ 2us) write_op.cc:260] PREPARE: finished
0804 14:52:37.733606 (+ 44us) write_op.cc:270] Start()
0804 14:52:38.411810 (+678204us) write_op.cc:276] Timestamp: P: 1785855157733546 usec, L: 0
0804 14:52:38.411822 (+ 12us) op_driver.cc:331] REPLICATION: starting
0804 14:52:38.412029 (+ 207us) log.cc:842] Serialized 354 byte log entry
0804 14:52:38.910806 (+498777us) op_driver.cc:444] REPLICATION: finished
0804 14:52:38.910965 (+ 159us) write_op.cc:301] APPLY: starting
0804 14:52:38.910984 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:38.911018 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:38.911034 (+ 16us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:38.911096 (+ 62us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:38.911134 (+ 38us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:38.911141 (+ 7us) write_op.cc:312] APPLY: finished
0804 14:52:38.911238 (+ 97us) log.cc:842] Serialized 22 byte log entry
0804 14:52:38.911480 (+ 242us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:38.911497 (+ 17us) write_op.cc:454] Released schema lock
0804 14:52:38.911502 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":103,"num_ops":1,"prepare.queue_time_us":748521,"prepare.run_cpu_time_us":709,"prepare.run_wall_time_us":678790,"replication_time_us":498961,"thread_start_us":58,"threads_started":1}]]}
I20260804 14:52:38.912662 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:52:38.912919 29575 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:39.010160 29582 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:39.092268 29582 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:39.092443 29582 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:39.093055 29582 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:52:39.093641 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:39.094094 29097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:52:39.095456 29185 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:52:39.095652 29582 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:39.352638 29575 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:39.352805 29575 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:39.353456 29575 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:52:39.353876 29272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:52:39.353974 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:39.354779 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:52:39.355017 29575 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:40.221534 29575 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:40.221719 29575 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:40.222332 29575 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:52:40.222805 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:40.222831 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:52:40.223466 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:52:40.223809 29575 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:40.224056 29604 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=4, attempt_no=0}) took 1310 ms. Trace:
I20260804 14:52:40.224175 29604 rpcz_store.cc:276] 0804 14:52:38.913213 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:38.913273 (+ 60us) service_pool.cc:227] Handling call
0804 14:52:40.224044 (+1310771us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:39.067132 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:52:39.067268 (+ 136us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:39.067272 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:52:39.067274 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:39.067359 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:39.067368 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:39.067370 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:39.067410 (+ 40us) tablet.cc:699] Row locks acquired
0804 14:52:39.067411 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:39.067447 (+ 36us) write_op.cc:270] Start()
0804 14:52:39.652652 (+585205us) write_op.cc:276] Timestamp: P: 1785855159067395 usec, L: 0
0804 14:52:39.652656 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:52:39.652846 (+ 190us) log.cc:842] Serialized 354 byte log entry
0804 14:52:40.222867 (+570021us) op_driver.cc:444] REPLICATION: finished
0804 14:52:40.223064 (+ 197us) write_op.cc:301] APPLY: starting
0804 14:52:40.223082 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:40.223116 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:40.223138 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:40.223241 (+ 103us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:40.223278 (+ 37us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:40.223283 (+ 5us) write_op.cc:312] APPLY: finished
0804 14:52:40.223356 (+ 73us) log.cc:842] Serialized 22 byte log entry
0804 14:52:40.223927 (+ 571us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:40.223943 (+ 16us) write_op.cc:454] Released schema lock
0804 14:52:40.223949 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":145,"num_ops":1,"prepare.queue_time_us":153479,"prepare.run_cpu_time_us":656,"prepare.run_wall_time_us":585776,"replication_time_us":570186,"thread_start_us":51,"threads_started":1}]]}
I20260804 14:52:40.617754 29599 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:40.617969 29599 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:40.618580 29599 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:52:40.619050 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:40.619429 29097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:52:40.619901 29185 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:52:40.626482 29575 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:40.626662 29575 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:40.627347 29575 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:52:40.627898 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:52:40.628016 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:40.628135 29604 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=5, attempt_no=0}) took 1531 ms. Trace:
I20260804 14:52:40.628212 29604 rpcz_store.cc:276] 0804 14:52:39.096689 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:39.096747 (+ 58us) service_pool.cc:227] Handling call
0804 14:52:40.628122 (+1531375us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:39.652945 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:52:39.653065 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:39.653069 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:52:39.653070 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:39.653140 (+ 70us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:39.653148 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:39.653150 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:39.653212 (+ 62us) tablet.cc:699] Row locks acquired
0804 14:52:39.653213 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:39.653257 (+ 44us) write_op.cc:270] Start()
0804 14:52:40.254413 (+601156us) write_op.cc:276] Timestamp: P: 1785855159653215 usec, L: 0
0804 14:52:40.254454 (+ 41us) op_driver.cc:331] REPLICATION: starting
0804 14:52:40.254706 (+ 252us) log.cc:842] Serialized 354 byte log entry
0804 14:52:40.627547 (+372841us) op_driver.cc:444] REPLICATION: finished
0804 14:52:40.627681 (+ 134us) write_op.cc:301] APPLY: starting
0804 14:52:40.627704 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:40.627740 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:40.627757 (+ 17us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:40.627830 (+ 73us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:40.627864 (+ 34us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:40.627869 (+ 5us) write_op.cc:312] APPLY: finished
0804 14:52:40.627947 (+ 78us) log.cc:842] Serialized 22 byte log entry
0804 14:52:40.628026 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:40.628040 (+ 14us) write_op.cc:454] Released schema lock
0804 14:52:40.628046 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":62,"num_ops":1,"prepare.queue_time_us":555806,"prepare.run_cpu_time_us":772,"prepare.run_wall_time_us":601855,"raft.queue_time_us":97,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":136,"replication_time_us":373066,"spinlock_wait_cycles":74496}]]}
I20260804 14:52:40.629132 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:52:40.629316 29575 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:40.850782 29599 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:41.555766 29599 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:41.555909 29599 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:41.556474 29599 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:52:41.556892 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:41.557049 29097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:52:41.557934 29185 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:52:41.558461 29615 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=6, attempt_no=0}) took 1332 ms. Trace:
I20260804 14:52:41.558529 29615 rpcz_store.cc:276] 0804 14:52:40.225711 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:40.225790 (+ 79us) service_pool.cc:227] Handling call
0804 14:52:41.558449 (+1332659us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:40.254850 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:52:40.254960 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:40.254965 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:52:40.254966 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:40.255057 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:40.255069 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:40.255074 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:40.255142 (+ 68us) tablet.cc:699] Row locks acquired
0804 14:52:40.255145 (+ 3us) write_op.cc:260] PREPARE: finished
0804 14:52:40.255193 (+ 48us) write_op.cc:270] Start()
0804 14:52:40.802376 (+547183us) write_op.cc:276] Timestamp: P: 1785855160255144 usec, L: 0
0804 14:52:40.802380 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:52:40.802635 (+ 255us) log.cc:842] Serialized 354 byte log entry
0804 14:52:41.557599 (+754964us) op_driver.cc:444] REPLICATION: finished
0804 14:52:41.557777 (+ 178us) write_op.cc:301] APPLY: starting
0804 14:52:41.557800 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:41.557834 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:41.557880 (+ 46us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:41.557948 (+ 68us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:41.557980 (+ 32us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:41.557985 (+ 5us) write_op.cc:312] APPLY: finished
0804 14:52:41.558092 (+ 107us) log.cc:842] Serialized 43 byte log entry
0804 14:52:41.558340 (+ 248us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:41.558353 (+ 13us) write_op.cc:454] Released schema lock
0804 14:52:41.558357 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":122,"num_ops":1,"prepare.queue_time_us":28599,"prepare.run_cpu_time_us":795,"prepare.run_wall_time_us":547853,"replication_time_us":755151,"thread_start_us":66,"threads_started":1}]]}
I20260804 14:52:41.637099 29575 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:41.637259 29575 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:41.637907 29575 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:52:41.638455 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:52:41.638695 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:41.639138 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:52:41.838209 29575 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:41.840090 29615 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=7, attempt_no=0}) took 1210 ms. Trace:
I20260804 14:52:41.840209 29615 rpcz_store.cc:276] 0804 14:52:40.629956 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:40.630065 (+ 109us) service_pool.cc:227] Handling call
0804 14:52:41.840075 (+1210010us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:40.802772 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:52:40.802927 (+ 155us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:40.802932 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:52:40.802934 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:40.803028 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:40.803041 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:40.803049 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:40.803089 (+ 40us) tablet.cc:699] Row locks acquired
0804 14:52:40.803090 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:40.803127 (+ 37us) write_op.cc:270] Start()
0804 14:52:41.140332 (+337205us) write_op.cc:276] Timestamp: P: 1785855160803079 usec, L: 0
0804 14:52:41.140336 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:52:41.140612 (+ 276us) log.cc:842] Serialized 354 byte log entry
0804 14:52:41.839507 (+698895us) op_driver.cc:444] REPLICATION: finished
0804 14:52:41.839611 (+ 104us) write_op.cc:301] APPLY: starting
0804 14:52:41.839631 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:41.839664 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:41.839681 (+ 17us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:41.839769 (+ 88us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:41.839805 (+ 36us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:41.839810 (+ 5us) write_op.cc:312] APPLY: finished
0804 14:52:41.839894 (+ 84us) log.cc:842] Serialized 22 byte log entry
0804 14:52:41.839979 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:41.839994 (+ 15us) write_op.cc:454] Released schema lock
0804 14:52:41.840000 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":29,"num_ops":1,"prepare.queue_time_us":172297,"prepare.run_cpu_time_us":833,"prepare.run_wall_time_us":337910,"replication_time_us":699127}]]}
I20260804 14:52:42.166400 29575 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:42.166604 29575 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:42.167245 29575 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:52:42.167945 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:52:42.168123 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:42.168711 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:52:42.169024 29575 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:42.487130 29599 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:42.675959 29575 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:42.676136 29575 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:42.676717 29575 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:52:42.677326 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:52:42.677395 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:42.678260 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:52:42.678452 29575 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:43.184545 29599 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:43.184712 29599 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:43.185226 29599 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:52:43.185947 29096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:52:43.186233 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:43.186760 29185 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:52:43.677944 29638 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:43.678129 29638 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:43.678753 29638 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:52:43.679421 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:52:43.679695 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:43.679800 29641 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=10, attempt_no=0}) took 1509 ms. Trace:
I20260804 14:52:43.679852 29641 rpcz_store.cc:276] 0804 14:52:42.170380 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:42.170482 (+ 102us) service_pool.cc:227] Handling call
0804 14:52:43.679765 (+1509283us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:42.623413 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:52:42.623528 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:42.623533 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:52:42.623534 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:42.623625 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:42.623636 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:42.623639 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:42.677806 (+ 54167us) tablet.cc:699] Row locks acquired
0804 14:52:42.677809 (+ 3us) write_op.cc:260] PREPARE: finished
0804 14:52:42.677861 (+ 52us) write_op.cc:270] Start()
0804 14:52:43.385087 (+707226us) write_op.cc:276] Timestamp: P: 1785855162677834 usec, L: 0
0804 14:52:43.385091 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:52:43.385314 (+ 223us) log.cc:842] Serialized 354 byte log entry
0804 14:52:43.678963 (+293649us) op_driver.cc:444] REPLICATION: finished
0804 14:52:43.679281 (+ 318us) write_op.cc:301] APPLY: starting
0804 14:52:43.679302 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:43.679362 (+ 60us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:43.679378 (+ 16us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:43.679450 (+ 72us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:43.679476 (+ 26us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:43.679480 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:52:43.679573 (+ 93us) log.cc:842] Serialized 43 byte log entry
0804 14:52:43.679651 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:43.679664 (+ 13us) write_op.cc:454] Released schema lock
0804 14:52:43.679671 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":255,"num_ops":1,"prepare.queue_time_us":452247,"prepare.run_cpu_time_us":936,"prepare.run_wall_time_us":761957,"replication_time_us":293840,"row_lock_wait_count":1,"row_lock_wait_us":54008,"spinlock_wait_cycles":30976,"thread_start_us":102,"threads_started":1}]]}
I20260804 14:52:43.680434 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:52:43.680809 29638 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:44.032070 29599 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:44.102277 29638 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:44.102474 29638 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:44.103112 29638 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:52:44.103583 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:44.103780 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:52:44.104064 29641 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=11, attempt_no=0}) took 1424 ms. Trace:
I20260804 14:52:44.104159 29641 rpcz_store.cc:276] 0804 14:52:42.679151 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:42.679231 (+ 80us) service_pool.cc:227] Handling call
0804 14:52:44.104050 (+1424819us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:43.385396 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:52:43.385521 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:43.385526 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:52:43.385527 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:43.385618 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:43.385628 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:43.385631 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:43.385682 (+ 51us) tablet.cc:699] Row locks acquired
0804 14:52:43.385683 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:43.385720 (+ 37us) write_op.cc:270] Start()
0804 14:52:43.722948 (+337228us) write_op.cc:276] Timestamp: P: 1785855163385756 usec, L: 0
0804 14:52:43.722954 (+ 6us) op_driver.cc:331] REPLICATION: starting
0804 14:52:43.723169 (+ 215us) log.cc:842] Serialized 354 byte log entry
0804 14:52:44.103468 (+380299us) op_driver.cc:444] REPLICATION: finished
0804 14:52:44.103588 (+ 120us) write_op.cc:301] APPLY: starting
0804 14:52:44.103609 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:44.103664 (+ 55us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:44.103678 (+ 14us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:44.103743 (+ 65us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:44.103775 (+ 32us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:44.103783 (+ 8us) write_op.cc:312] APPLY: finished
0804 14:52:44.103863 (+ 80us) log.cc:842] Serialized 22 byte log entry
0804 14:52:44.103942 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:44.103965 (+ 23us) write_op.cc:454] Released schema lock
0804 14:52:44.103970 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":73,"num_ops":1,"prepare.queue_time_us":705794,"prepare.run_cpu_time_us":837,"prepare.run_wall_time_us":337886,"replication_time_us":380484}]]}
I20260804 14:52:44.104348 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:52:44.104878 29638 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:44.711766 29638 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:44.711933 29638 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:44.712518 29638 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:52:44.712929 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:44.713071 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:52:44.713704 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:52:44.713873 29638 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:44.714062 29649 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=12, attempt_no=0}) took 1032 ms. Trace:
I20260804 14:52:44.714124 29649 rpcz_store.cc:276] 0804 14:52:43.681707 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:43.681794 (+ 87us) service_pool.cc:227] Handling call
0804 14:52:44.714045 (+1032251us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:43.723329 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:52:43.723486 (+ 157us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:43.723491 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:52:43.723493 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:43.723584 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:43.723597 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:43.723602 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:44.104646 (+381044us) lock_manager.cc:599] Waited 380914us for lock on <redacted> (key hash 2724938281930601886)
0804 14:52:44.104654 (+ 8us) tablet.cc:699] Row locks acquired
0804 14:52:44.104656 (+ 2us) write_op.cc:260] PREPARE: finished
0804 14:52:44.104698 (+ 42us) write_op.cc:270] Start()
0804 14:52:44.334917 (+230219us) write_op.cc:276] Timestamp: P: 1785855164104674 usec, L: 0
0804 14:52:44.334922 (+ 5us) op_driver.cc:331] REPLICATION: starting
0804 14:52:44.335114 (+ 192us) log.cc:842] Serialized 354 byte log entry
0804 14:52:44.713126 (+378012us) op_driver.cc:444] REPLICATION: finished
0804 14:52:44.713337 (+ 211us) write_op.cc:301] APPLY: starting
0804 14:52:44.713367 (+ 30us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:44.713416 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:44.713418 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:44.713481 (+ 63us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:44.713507 (+ 26us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:44.713511 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:52:44.713656 (+ 145us) log.cc:842] Serialized 43 byte log entry
0804 14:52:44.713727 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:44.713743 (+ 16us) write_op.cc:454] Released schema lock
0804 14:52:44.713749 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":152,"num_ops":1,"prepare.queue_time_us":41109,"prepare.run_cpu_time_us":881,"prepare.run_wall_time_us":611866,"replication_time_us":378180,"row_lock_wait_count":1,"row_lock_wait_us":380914,"thread_start_us":74,"threads_started":1}]]}
I20260804 14:52:44.956892 29599 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:44.957036 29599 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:44.957535 29599 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:52:44.957892 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:44.957876 29096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:52:44.958508 29185 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:52:45.302465 29655 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:45.302688 29655 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:45.303299 29655 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:52:45.303689 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:45.303794 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:52:45.304199 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:52:45.304454 29655 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:45.305115 29660 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=13, attempt_no=0}) took 1199 ms. Trace:
I20260804 14:52:45.305223 29660 rpcz_store.cc:276] 0804 14:52:44.105772 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:44.105825 (+ 53us) service_pool.cc:227] Handling call
0804 14:52:45.305099 (+1199274us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:44.335252 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:52:44.335395 (+ 143us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:44.335399 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:52:44.335401 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:44.335486 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:44.335498 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:44.335500 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:44.335549 (+ 49us) tablet.cc:699] Row locks acquired
0804 14:52:44.335550 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:44.335599 (+ 49us) write_op.cc:270] Start()
0804 14:52:45.228774 (+893175us) write_op.cc:276] Timestamp: P: 1785855164335556 usec, L: 0
0804 14:52:45.228778 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:52:45.228954 (+ 176us) log.cc:842] Serialized 354 byte log entry
0804 14:52:45.304475 (+ 75521us) op_driver.cc:444] REPLICATION: finished
0804 14:52:45.304729 (+ 254us) write_op.cc:301] APPLY: starting
0804 14:52:45.304743 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:45.304767 (+ 24us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:45.304782 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:45.304833 (+ 51us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:45.304860 (+ 27us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:45.304864 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:52:45.304921 (+ 57us) log.cc:842] Serialized 22 byte log entry
0804 14:52:45.305010 (+ 89us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:45.305021 (+ 11us) write_op.cc:454] Released schema lock
0804 14:52:45.305027 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":197,"num_ops":1,"prepare.queue_time_us":228993,"prepare.run_cpu_time_us":743,"prepare.run_wall_time_us":893837,"replication_time_us":75139,"thread_start_us":110,"threads_started":1}]]}
I20260804 14:52:45.632215 29655 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:45.632365 29655 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:45.632911 29655 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:52:45.633610 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:52:45.633610 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:45.634269 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:52:45.634931 29655 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:45.693934 29599 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:46.153784 29655 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:46.153977 29655 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:46.154577 29655 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:52:46.155237 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:52:46.155339 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:46.155833 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:52:46.156214 29655 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:46.677814 29599 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:46.677960 29599 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:46.678480 29599 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:52:46.679409 29096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:52:46.679411 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:46.680023 29185 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:52:47.098652 29599 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:47.727438 29675 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:47.727618 29675 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:47.728324 29675 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:52:47.728734 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:47.728911 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:52:47.729408 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:52:47.729561 29682 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=16, attempt_no=0}) took 2093 ms. Trace:
I20260804 14:52:47.729631 29682 rpcz_store.cc:276] 0804 14:52:45.635775 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:45.635838 (+ 63us) service_pool.cc:227] Handling call
0804 14:52:47.729547 (+2093709us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:45.944041 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:52:45.944221 (+ 180us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:45.944226 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:52:45.944227 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:45.944324 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:45.944335 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:45.944337 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:45.944412 (+ 75us) tablet.cc:699] Row locks acquired
0804 14:52:45.944415 (+ 3us) write_op.cc:260] PREPARE: finished
0804 14:52:45.944459 (+ 44us) write_op.cc:270] Start()
0804 14:52:46.864672 (+920213us) write_op.cc:276] Timestamp: P: 1785855165944383 usec, L: 0
0804 14:52:46.864678 (+ 6us) op_driver.cc:331] REPLICATION: starting
0804 14:52:46.864926 (+ 248us) log.cc:842] Serialized 354 byte log entry
0804 14:52:47.728911 (+863985us) op_driver.cc:444] REPLICATION: finished
0804 14:52:47.729092 (+ 181us) write_op.cc:301] APPLY: starting
0804 14:52:47.729112 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:47.729147 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:47.729160 (+ 13us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:47.729246 (+ 86us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:47.729276 (+ 30us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:47.729280 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:52:47.729362 (+ 82us) log.cc:842] Serialized 43 byte log entry
0804 14:52:47.729430 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:47.729447 (+ 17us) write_op.cc:454] Released schema lock
0804 14:52:47.729452 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":130,"num_ops":1,"prepare.queue_time_us":307834,"prepare.run_cpu_time_us":922,"prepare.run_wall_time_us":921022,"replication_time_us":864210,"thread_start_us":72,"threads_started":1}]]}
I20260804 14:52:48.403088 29599 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:48.403337 29599 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:48.403587 29272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.24.6.131:48343 (request call id 1850) took 1303 ms. Trace:
I20260804 14:52:48.404037 29599 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:52:48.404557 29097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:52:48.404575 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:48.403682 29272 rpcz_store.cc:276] 0804 14:52:47.100159 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:47.100230 (+ 71us) service_pool.cc:227] Handling call
0804 14:52:47.100387 (+ 157us) raft_consensus.cc:1457] Updating replica for 2 ops
0804 14:52:47.100451 (+ 64us) raft_consensus.cc:1520] Early marking committed up to index 0
0804 14:52:47.100453 (+ 2us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0804 14:52:47.100548 (+ 95us) write_op.cc:270] Start()
0804 14:52:47.716724 (+616176us) write_op.cc:276] Timestamp: P: 1785855165944383 usec, L: 0
0804 14:52:47.717129 (+ 405us) write_op.cc:270] Start()
0804 14:52:48.402347 (+685218us) write_op.cc:276] Timestamp: P: 1785855166865390 usec, L: 0
0804 14:52:48.402843 (+ 496us) log.cc:842] Serialized 708 byte log entry
0804 14:52:48.402969 (+ 126us) raft_consensus.cc:1631] Marking committed up to 17
0804 14:52:48.402975 (+ 6us) raft_consensus.cc:1683] Filling consensus response to leader.
0804 14:52:48.403012 (+ 37us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0804 14:52:48.403501 (+ 489us) raft_consensus.cc:1671] finished
0804 14:52:48.403503 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0804 14:52:48.403571 (+ 68us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:47.717112 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:52:47.717211 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:47.717215 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:52:47.717216 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:47.717287 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:47.717297 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:47.717299 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:47.717336 (+ 37us) tablet.cc:699] Row locks acquired
0804 14:52:47.717337 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0804 14:52:48.402800 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:52:48.402906 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:48.402910 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:52:48.402912 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:48.402984 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:48.402994 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:48.402996 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:48.403078 (+ 82us) tablet.cc:699] Row locks acquired
0804 14:52:48.403080 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":74,"threads_started":1,"wal-append.queue_time_us":135,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":4,"prepare.queue_time_us":240,"prepare.run_cpu_time_us":309,"prepare.run_wall_time_us":310,"thread_start_us":111,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":298,"prepare.run_cpu_time_us":380,"prepare.run_wall_time_us":380,"thread_start_us":91,"threads_started":1}]]}
I20260804 14:52:48.405049 29187 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:52:48.405246 29599 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:48.406599 29689 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=17, attempt_no=0}) took 2248 ms. Trace:
I20260804 14:52:48.406740 29689 rpcz_store.cc:276] 0804 14:52:46.157926 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:46.157971 (+ 45us) service_pool.cc:227] Handling call
0804 14:52:48.406582 (+2248611us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:46.865118 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:52:46.865235 (+ 117us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:46.865240 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:52:46.865241 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:46.865328 (+ 87us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:46.865342 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:46.865345 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:46.865394 (+ 49us) tablet.cc:699] Row locks acquired
0804 14:52:46.865395 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:46.865443 (+ 48us) write_op.cc:270] Start()
0804 14:52:46.898639 (+ 33196us) write_op.cc:276] Timestamp: P: 1785855166865390 usec, L: 0
0804 14:52:46.898644 (+ 5us) op_driver.cc:331] REPLICATION: starting
0804 14:52:46.898906 (+ 262us) log.cc:842] Serialized 354 byte log entry
0804 14:52:48.405778 (+1506872us) op_driver.cc:444] REPLICATION: finished
0804 14:52:48.406035 (+ 257us) write_op.cc:301] APPLY: starting
0804 14:52:48.406100 (+ 65us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:48.406136 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:48.406155 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:48.406224 (+ 69us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:48.406247 (+ 23us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:48.406250 (+ 3us) write_op.cc:312] APPLY: finished
0804 14:52:48.406354 (+ 104us) log.cc:842] Serialized 43 byte log entry
0804 14:52:48.406461 (+ 107us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:48.406475 (+ 14us) write_op.cc:454] Released schema lock
0804 14:52:48.406481 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":204,"num_ops":1,"prepare.queue_time_us":706781,"prepare.run_cpu_time_us":823,"prepare.run_wall_time_us":33911,"replication_time_us":1507108,"thread_start_us":135,"threads_started":1}]]}
I20260804 14:52:48.460848 29675 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:48.876144 29675 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:48.876310 29675 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:48.876797 29675 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:52:48.877170 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:48.877321 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:52:48.877702 29689 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=18, attempt_no=0}) took 1146 ms. Trace:
I20260804 14:52:48.877759 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:52:48.877938 29675 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:48.877781 29689 rpcz_store.cc:276] 0804 14:52:47.731165 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:47.731260 (+ 95us) service_pool.cc:227] Handling call
0804 14:52:48.877689 (+1146429us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:47.731765 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:52:47.731845 (+ 80us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:47.731847 (+ 2us) write_op.cc:435] Acquired schema lock
0804 14:52:47.731848 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:47.731912 (+ 64us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:47.731919 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:47.731922 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:47.731955 (+ 33us) tablet.cc:699] Row locks acquired
0804 14:52:47.731956 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:47.731986 (+ 30us) write_op.cc:270] Start()
0804 14:52:48.669156 (+937170us) write_op.cc:276] Timestamp: P: 1785855167731928 usec, L: 0
0804 14:52:48.669160 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:52:48.669350 (+ 190us) log.cc:842] Serialized 354 byte log entry
0804 14:52:48.877158 (+207808us) op_driver.cc:444] REPLICATION: finished
0804 14:52:48.877249 (+ 91us) write_op.cc:301] APPLY: starting
0804 14:52:48.877271 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:48.877305 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:48.877320 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:48.877394 (+ 74us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:48.877423 (+ 29us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:48.877430 (+ 7us) write_op.cc:312] APPLY: finished
0804 14:52:48.877513 (+ 83us) log.cc:842] Serialized 43 byte log entry
0804 14:52:48.877576 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:48.877589 (+ 13us) write_op.cc:454] Released schema lock
0804 14:52:48.877595 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":32,"mutex_wait_us":30,"num_ops":1,"prepare.queue_time_us":161,"prepare.run_cpu_time_us":647,"prepare.run_wall_time_us":937728,"raft.queue_time_us":91,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":244,"replication_time_us":207972,"spinlock_wait_cycles":110720,"thread_start_us":77,"threads_started":1}]]}
I20260804 14:52:49.207312 29675 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:49.207484 29675 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:49.208067 29675 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:52:49.208405 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:49.208508 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:52:49.209208 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:52:49.209591 29675 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:49.668318 29599 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:49.668500 29599 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:49.669171 29599 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:52:49.669589 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:49.669687 29097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:52:49.670156 29185 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:52:49.690326 29675 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:49.690485 29675 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:49.691177 29675 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:52:49.691666 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:49.692088 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:52:49.692531 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:52:49.692839 29675 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:50.371752 29700 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:50.371960 29700 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:50.372587 29700 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:52:50.376587 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:50.377135 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:52:50.379104 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:52:50.379488 29700 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:50.877482 29700 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:50.877647 29700 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:50.878242 29700 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:52:50.878682 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:50.878782 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:52:50.879252 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:52:50.879379 29705 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=21, attempt_no=0}) took 1669 ms. Trace:
I20260804 14:52:50.879552 29700 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:50.879458 29705 rpcz_store.cc:276] 0804 14:52:49.210139 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:49.210210 (+ 71us) service_pool.cc:227] Handling call
0804 14:52:50.879362 (+1669152us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:49.652523 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:52:49.652658 (+ 135us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:49.652664 (+ 6us) write_op.cc:435] Acquired schema lock
0804 14:52:49.652666 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:49.652752 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:49.652765 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:49.652771 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:49.652819 (+ 48us) tablet.cc:699] Row locks acquired
0804 14:52:49.652820 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:49.652871 (+ 51us) write_op.cc:270] Start()
0804 14:52:50.568045 (+915174us) write_op.cc:276] Timestamp: P: 1785855169652793 usec, L: 0
0804 14:52:50.568049 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:52:50.568228 (+ 179us) log.cc:842] Serialized 354 byte log entry
0804 14:52:50.878569 (+310341us) op_driver.cc:444] REPLICATION: finished
0804 14:52:50.878821 (+ 252us) write_op.cc:301] APPLY: starting
0804 14:52:50.878840 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:50.878877 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:50.878897 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:50.879069 (+ 172us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:50.879111 (+ 42us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:50.879115 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:52:50.879188 (+ 73us) log.cc:842] Serialized 22 byte log entry
0804 14:52:50.879268 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:50.879281 (+ 13us) write_op.cc:454] Released schema lock
0804 14:52:50.879287 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":204,"num_ops":1,"prepare.queue_time_us":441925,"prepare.run_cpu_time_us":727,"prepare.run_wall_time_us":915794,"replication_time_us":310499,"thread_start_us":83,"threads_started":1}]]}
I20260804 14:52:50.901290 29599 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:50.901652 29272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.24.6.131:48343 (request call id 2065) took 1231 ms. Trace:
I20260804 14:52:50.901736 29272 rpcz_store.cc:276] 0804 14:52:49.670068 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:49.670137 (+ 69us) service_pool.cc:227] Handling call
0804 14:52:49.670295 (+ 158us) raft_consensus.cc:1457] Updating replica for 2 ops
0804 14:52:49.670380 (+ 85us) raft_consensus.cc:1520] Early marking committed up to index 20
0804 14:52:49.670524 (+ 144us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0804 14:52:49.670622 (+ 98us) write_op.cc:270] Start()
0804 14:52:50.545855 (+875233us) write_op.cc:276] Timestamp: P: 1785855168669816 usec, L: 0
0804 14:52:50.546286 (+ 431us) write_op.cc:270] Start()
0804 14:52:50.900521 (+354235us) write_op.cc:276] Timestamp: P: 1785855168880007 usec, L: 0
0804 14:52:50.900824 (+ 303us) log.cc:842] Serialized 708 byte log entry
0804 14:52:50.901039 (+ 215us) raft_consensus.cc:1631] Marking committed up to 21
0804 14:52:50.901217 (+ 178us) raft_consensus.cc:1683] Filling consensus response to leader.
0804 14:52:50.901243 (+ 26us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0804 14:52:50.901590 (+ 347us) raft_consensus.cc:1671] finished
0804 14:52:50.901592 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0804 14:52:50.901635 (+ 43us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:50.546311 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:52:50.546497 (+ 186us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:50.546502 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:52:50.546504 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:50.546659 (+ 155us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:50.546671 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:50.546674 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:50.546720 (+ 46us) tablet.cc:699] Row locks acquired
0804 14:52:50.546721 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:50.901078 (+354357us) op_driver.cc:444] REPLICATION: finished
0804 14:52:50.901260 (+ 182us) write_op.cc:301] APPLY: starting
0804 14:52:50.901275 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:50.901305 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:50.901307 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:50.901361 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:50.901387 (+ 26us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:50.901405 (+ 18us) write_op.cc:312] APPLY: finished
0804 14:52:50.901464 (+ 59us) log.cc:842] Serialized 22 byte log entry
0804 14:52:50.901516 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:50.901604 (+ 88us) write_op.cc:454] Released schema lock
0804 14:52:50.901610 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0804 14:52:50.901680 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:50.901682 (+ 2us) write_op.cc:454] Released schema lock
Related trace 'op':
0804 14:52:50.900778 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:52:50.900893 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:50.900900 (+ 7us) write_op.cc:435] Acquired schema lock
0804 14:52:50.900903 (+ 3us) tablet.cc:660] Decoding operations
0804 14:52:50.901022 (+ 119us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:50.901036 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:50.901039 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:50.901088 (+ 49us) tablet.cc:699] Row locks acquired
0804 14:52:50.901089 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"spinlock_wait_cycles":384,"thread_start_us":151,"threads_started":1,"wal-append.queue_time_us":243,"child_traces":[["op",{"apply.queue_time_us":137,"apply.run_cpu_time_us":405,"apply.run_wall_time_us":463,"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":260,"prepare.run_cpu_time_us":499,"prepare.run_wall_time_us":498,"replication_time_us":1230438,"thread_start_us":184,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":105,"prepare.run_cpu_time_us":418,"prepare.run_wall_time_us":417}]]}
I20260804 14:52:51.173323 29599 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:51.173542 29599 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:51.174286 29599 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:52:51.174813 29097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:52:51.174867 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:51.175340 29187 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:52:51.278872 29700 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:51.279038 29700 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:51.279789 29700 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:52:51.280332 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:52:51.280364 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:51.280437 29705 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=22, attempt_no=0}) took 1587 ms. Trace:
I20260804 14:52:51.280516 29705 rpcz_store.cc:276] 0804 14:52:49.693194 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:49.693258 (+ 64us) service_pool.cc:227] Handling call
0804 14:52:51.280425 (+1587167us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:50.568351 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:52:50.568459 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:50.568464 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:52:50.568465 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:50.568547 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:50.568557 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:50.568560 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:50.568606 (+ 46us) tablet.cc:699] Row locks acquired
0804 14:52:50.568608 (+ 2us) write_op.cc:260] PREPARE: finished
0804 14:52:50.568654 (+ 46us) write_op.cc:270] Start()
0804 14:52:51.038857 (+470203us) write_op.cc:276] Timestamp: P: 1785855170568579 usec, L: 0
0804 14:52:51.038861 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:52:51.039089 (+ 228us) log.cc:842] Serialized 354 byte log entry
0804 14:52:51.279871 (+240782us) op_driver.cc:444] REPLICATION: finished
0804 14:52:51.279974 (+ 103us) write_op.cc:301] APPLY: starting
0804 14:52:51.280021 (+ 47us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:51.280055 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:51.280074 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:51.280164 (+ 90us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:51.280196 (+ 32us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:51.280202 (+ 6us) write_op.cc:312] APPLY: finished
0804 14:52:51.280273 (+ 71us) log.cc:842] Serialized 22 byte log entry
0804 14:52:51.280330 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:51.280343 (+ 13us) write_op.cc:454] Released schema lock
0804 14:52:51.280351 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":36,"num_ops":1,"prepare.queue_time_us":874803,"prepare.run_cpu_time_us":968,"prepare.run_wall_time_us":471051,"raft.queue_time_us":77,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":240981,"thread_start_us":182,"threads_started":1}]]}
I20260804 14:52:51.281301 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:52:51.443548 29700 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:51.714214 29599 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:52.093521 29599 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:52.093685 29599 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:52.094216 29599 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:52:52.094671 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:52.094907 29097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:52:52.095563 29185 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:52:52.126432 29714 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:52.126613 29714 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:52.127322 29714 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:52:52.127907 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:52:52.127951 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:52.128611 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:52:52.516218 29599 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:52.847389 29714 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:52.849248 29722 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=25, attempt_no=0}) took 1402 ms. Trace:
I20260804 14:52:52.849368 29722 rpcz_store.cc:276] 0804 14:52:51.446919 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:51.446998 (+ 79us) service_pool.cc:227] Handling call
0804 14:52:52.849229 (+1402231us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:51.941053 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:52:51.941146 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:51.941150 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:52:51.941151 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:51.941219 (+ 68us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:51.941227 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:51.941229 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:51.941266 (+ 37us) tablet.cc:699] Row locks acquired
0804 14:52:51.941267 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:51.941303 (+ 36us) write_op.cc:270] Start()
0804 14:52:52.096537 (+155234us) write_op.cc:276] Timestamp: P: 1785855171941273 usec, L: 0
0804 14:52:52.096541 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:52:52.096693 (+ 152us) log.cc:842] Serialized 354 byte log entry
0804 14:52:52.848486 (+751793us) op_driver.cc:444] REPLICATION: finished
0804 14:52:52.848723 (+ 237us) write_op.cc:301] APPLY: starting
0804 14:52:52.848743 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:52.848780 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:52.848802 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:52.848869 (+ 67us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:52.848901 (+ 32us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:52.848904 (+ 3us) write_op.cc:312] APPLY: finished
0804 14:52:52.849023 (+ 119us) log.cc:842] Serialized 43 byte log entry
0804 14:52:52.849106 (+ 83us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:52.849129 (+ 23us) write_op.cc:454] Released schema lock
0804 14:52:52.849134 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":179,"num_ops":1,"prepare.queue_time_us":493621,"prepare.run_cpu_time_us":531,"prepare.run_wall_time_us":155692,"replication_time_us":751919,"thread_start_us":81,"threads_started":1}]]}
I20260804 14:52:53.590428 29714 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:53.590610 29714 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:53.591163 29714 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:52:53.591606 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:53.591657 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:52:53.592147 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:52:53.592686 29734 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=26, attempt_no=0}) took 1463 ms. Trace:
I20260804 14:52:53.592796 29734 rpcz_store.cc:276] 0804 14:52:52.129721 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:52.129785 (+ 64us) service_pool.cc:227] Handling call
0804 14:52:53.592663 (+1462878us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:52.130304 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:52:52.130405 (+ 101us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:52.130408 (+ 3us) write_op.cc:435] Acquired schema lock
0804 14:52:52.130409 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:52.130478 (+ 69us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:52.130485 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:52.130487 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:52.130521 (+ 34us) tablet.cc:699] Row locks acquired
0804 14:52:52.130522 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:52.130553 (+ 31us) write_op.cc:270] Start()
0804 14:52:52.311735 (+181182us) write_op.cc:276] Timestamp: P: 1785855172130505 usec, L: 0
0804 14:52:52.311739 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:52:52.312002 (+ 263us) log.cc:842] Serialized 354 byte log entry
0804 14:52:53.591711 (+1279709us) op_driver.cc:444] REPLICATION: finished
0804 14:52:53.592006 (+ 295us) write_op.cc:301] APPLY: starting
0804 14:52:53.592040 (+ 34us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:53.592076 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:53.592093 (+ 17us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:53.592157 (+ 64us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:53.592354 (+ 197us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:53.592359 (+ 5us) write_op.cc:312] APPLY: finished
0804 14:52:53.592444 (+ 85us) log.cc:842] Serialized 43 byte log entry
0804 14:52:53.592534 (+ 90us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:53.592547 (+ 13us) write_op.cc:454] Released schema lock
0804 14:52:53.592554 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":242,"num_ops":1,"prepare.queue_time_us":63,"prepare.run_cpu_time_us":698,"prepare.run_wall_time_us":181778,"replication_time_us":1279950,"thread_start_us":68,"threads_started":1}]]}
I20260804 14:52:53.625106 29599 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:53.625279 29599 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:53.625684 29272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.24.6.131:48343 (request call id 2070) took 1107 ms. Trace:
I20260804 14:52:53.625859 29599 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:52:53.625783 29272 rpcz_store.cc:276] 0804 14:52:52.518033 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:52.518129 (+ 96us) service_pool.cc:227] Handling call
0804 14:52:52.518291 (+ 162us) raft_consensus.cc:1457] Updating replica for 2 ops
0804 14:52:52.518357 (+ 66us) raft_consensus.cc:1520] Early marking committed up to index 26
0804 14:52:52.518430 (+ 73us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0804 14:52:52.518492 (+ 62us) write_op.cc:270] Start()
0804 14:52:52.904703 (+386211us) write_op.cc:276] Timestamp: P: 1785855171941273 usec, L: 0
0804 14:52:52.905028 (+ 325us) write_op.cc:270] Start()
0804 14:52:53.624278 (+719250us) write_op.cc:276] Timestamp: P: 1785855172130505 usec, L: 0
0804 14:52:53.624837 (+ 559us) log.cc:842] Serialized 708 byte log entry
0804 14:52:53.624984 (+ 147us) raft_consensus.cc:1631] Marking committed up to 26
0804 14:52:53.624991 (+ 7us) raft_consensus.cc:1683] Filling consensus response to leader.
0804 14:52:53.625029 (+ 38us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0804 14:52:53.625605 (+ 576us) raft_consensus.cc:1671] finished
0804 14:52:53.625607 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0804 14:52:53.625663 (+ 56us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:52.904980 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:52:52.905088 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:52.905092 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:52:52.905094 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:52.905165 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:52.905176 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:52.905178 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:52.905217 (+ 39us) tablet.cc:699] Row locks acquired
0804 14:52:52.905217 (+ 0us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0804 14:52:53.624709 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:52:53.624834 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:53.624839 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:52:53.624840 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:53.624938 (+ 98us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:53.624954 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:53.624957 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:53.625008 (+ 51us) tablet.cc:699] Row locks acquired
0804 14:52:53.625010 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":88,"threads_started":1,"wal-append.queue_time_us":296,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":90,"prepare.run_cpu_time_us":341,"prepare.run_wall_time_us":343}],["op",{"num_ops":1,"partition_lock_wait_us":8,"prepare.queue_time_us":294,"prepare.run_cpu_time_us":408,"prepare.run_wall_time_us":408,"thread_start_us":142,"threads_started":1}]]}
I20260804 14:52:53.626434 29097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:52:53.626426 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:53.627022 29187 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:52:53.627239 29599 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:53.925235 29599 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:53.925408 29599 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:53.926036 29599 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:52:53.926424 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:53.926615 29097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:52:53.926944 29734 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=27, attempt_no=0}) took 1075 ms. Trace:
I20260804 14:52:53.927026 29734 rpcz_store.cc:276] 0804 14:52:52.851083 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:52.851177 (+ 94us) service_pool.cc:227] Handling call
0804 14:52:53.926931 (+1075754us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:52.851813 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:52:52.851899 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:52.851902 (+ 3us) write_op.cc:435] Acquired schema lock
0804 14:52:52.851903 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:52.851967 (+ 64us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:52.851974 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:52.851976 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:52.852056 (+ 80us) tablet.cc:699] Row locks acquired
0804 14:52:52.852057 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:52.852089 (+ 32us) write_op.cc:270] Start()
0804 14:52:53.099359 (+247270us) write_op.cc:276] Timestamp: P: 1785855172852064 usec, L: 0
0804 14:52:53.099364 (+ 5us) op_driver.cc:331] REPLICATION: starting
0804 14:52:53.099613 (+ 249us) log.cc:842] Serialized 354 byte log entry
0804 14:52:53.926400 (+826787us) op_driver.cc:444] REPLICATION: finished
0804 14:52:53.926515 (+ 115us) write_op.cc:301] APPLY: starting
0804 14:52:53.926537 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:53.926569 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:53.926584 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:53.926648 (+ 64us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:53.926680 (+ 32us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:53.926684 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:52:53.926755 (+ 71us) log.cc:842] Serialized 22 byte log entry
0804 14:52:53.926827 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:53.926843 (+ 16us) write_op.cc:454] Released schema lock
0804 14:52:53.926847 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":60,"num_ops":1,"prepare.queue_time_us":166,"prepare.run_cpu_time_us":695,"prepare.run_wall_time_us":247865,"replication_time_us":827011,"thread_start_us":66,"threads_started":1}]]}
I20260804 14:52:53.927356 29185 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:52:53.927641 29599 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:54.266564 29714 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:54.926611 29714 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:54.926782 29714 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:54.927423 29714 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:52:54.927831 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:54.927976 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:52:54.928436 29747 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=28, attempt_no=0}) took 1334 ms. Trace:
I20260804 14:52:54.928593 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:52:54.928560 29747 rpcz_store.cc:276] 0804 14:52:53.594283 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:53.594390 (+ 107us) service_pool.cc:227] Handling call
0804 14:52:54.928417 (+1334027us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:53.594919 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:52:53.595016 (+ 97us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:53.595019 (+ 3us) write_op.cc:435] Acquired schema lock
0804 14:52:53.595024 (+ 5us) tablet.cc:660] Decoding operations
0804 14:52:53.595088 (+ 64us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:53.595094 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:53.595099 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:53.595143 (+ 44us) tablet.cc:699] Row locks acquired
0804 14:52:53.595144 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:53.595174 (+ 30us) write_op.cc:270] Start()
0804 14:52:54.450330 (+855156us) write_op.cc:276] Timestamp: P: 1785855173595123 usec, L: 0
0804 14:52:54.450334 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:52:54.450585 (+ 251us) log.cc:842] Serialized 354 byte log entry
0804 14:52:54.927642 (+477057us) op_driver.cc:444] REPLICATION: finished
0804 14:52:54.927888 (+ 246us) write_op.cc:301] APPLY: starting
0804 14:52:54.927908 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:54.927944 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:54.927960 (+ 16us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:54.928092 (+ 132us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:54.928127 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:54.928132 (+ 5us) write_op.cc:312] APPLY: finished
0804 14:52:54.928228 (+ 96us) log.cc:842] Serialized 43 byte log entry
0804 14:52:54.928290 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:54.928305 (+ 15us) write_op.cc:454] Released schema lock
0804 14:52:54.928310 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":178,"num_ops":1,"prepare.queue_time_us":74,"prepare.run_cpu_time_us":684,"prepare.run_wall_time_us":855740,"replication_time_us":477282,"thread_start_us":87,"threads_started":1}]]}
I20260804 14:52:54.928875 29714 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:55.258042 29745 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:55.258220 29745 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:55.259101 29745 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:52:55.259449 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:55.259644 29097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:52:55.260527 29185 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:52:55.385402 29714 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:55.385617 29714 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:55.386198 29714 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:52:55.386788 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:55.386795 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:52:55.386961 29747 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=29, attempt_no=0}) took 1458 ms. Trace:
I20260804 14:52:55.387022 29747 rpcz_store.cc:276] 0804 14:52:53.928444 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:53.928511 (+ 67us) service_pool.cc:227] Handling call
0804 14:52:55.386951 (+1458440us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:54.450694 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:52:54.450814 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:54.450819 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:52:54.450821 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:54.450907 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:54.450917 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:54.450920 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:54.450965 (+ 45us) tablet.cc:699] Row locks acquired
0804 14:52:54.450969 (+ 4us) write_op.cc:260] PREPARE: finished
0804 14:52:54.451016 (+ 47us) write_op.cc:270] Start()
0804 14:52:55.247183 (+796167us) write_op.cc:276] Timestamp: P: 1785855174450974 usec, L: 0
0804 14:52:55.247187 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:52:55.247432 (+ 245us) log.cc:842] Serialized 354 byte log entry
0804 14:52:55.386371 (+138939us) op_driver.cc:444] REPLICATION: finished
0804 14:52:55.386502 (+ 131us) write_op.cc:301] APPLY: starting
0804 14:52:55.386530 (+ 28us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:55.386565 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:55.386600 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:55.386672 (+ 72us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:55.386701 (+ 29us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:55.386705 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:52:55.386787 (+ 82us) log.cc:842] Serialized 43 byte log entry
0804 14:52:55.386851 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:55.386865 (+ 14us) write_op.cc:454] Released schema lock
0804 14:52:55.386870 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":67,"num_ops":1,"prepare.queue_time_us":521881,"prepare.run_cpu_time_us":734,"prepare.run_wall_time_us":796812,"replication_time_us":139156}]]}
I20260804 14:52:55.387338 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:52:55.585570 29714 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:55.869158 29745 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:56.037694 29745 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:56.037861 29745 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:56.038393 29745 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:52:56.038885 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:56.038888 29096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:52:56.039577 29185 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:52:56.039829 29745 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:56.223551 29745 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:56.223762 29745 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:56.224224 29745 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:52:56.224753 29097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:52:56.224807 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:56.225416 29187 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:52:56.225646 29745 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:56.949138 29759 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:56.949308 29759 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:56.949832 29759 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:52:56.950367 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:52:56.950367 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:56.951061 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:52:56.951339 29759 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:56.951566 29766 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=31, attempt_no=0}) took 1562 ms. Trace:
I20260804 14:52:56.951648 29766 rpcz_store.cc:276] 0804 14:52:55.388645 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:55.388707 (+ 62us) service_pool.cc:227] Handling call
0804 14:52:56.951546 (+1562839us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:55.389150 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:52:55.389244 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:55.389247 (+ 3us) write_op.cc:435] Acquired schema lock
0804 14:52:55.389248 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:55.389311 (+ 63us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:55.389318 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:55.389320 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:55.389352 (+ 32us) tablet.cc:699] Row locks acquired
0804 14:52:55.389353 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:55.389383 (+ 30us) write_op.cc:270] Start()
0804 14:52:56.377556 (+988173us) write_op.cc:276] Timestamp: P: 1785855175389354 usec, L: 0
0804 14:52:56.377561 (+ 5us) op_driver.cc:331] REPLICATION: starting
0804 14:52:56.377772 (+ 211us) log.cc:842] Serialized 354 byte log entry
0804 14:52:56.950743 (+572971us) op_driver.cc:444] REPLICATION: finished
0804 14:52:56.951004 (+ 261us) write_op.cc:301] APPLY: starting
0804 14:52:56.951027 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:56.951062 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:56.951075 (+ 13us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:56.951142 (+ 67us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:56.951174 (+ 32us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:56.951179 (+ 5us) write_op.cc:312] APPLY: finished
0804 14:52:56.951298 (+ 119us) log.cc:842] Serialized 43 byte log entry
0804 14:52:56.951357 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:56.951375 (+ 18us) write_op.cc:454] Released schema lock
0804 14:52:56.951382 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":198,"num_ops":1,"prepare.queue_time_us":53,"prepare.run_cpu_time_us":649,"prepare.run_wall_time_us":988705,"replication_time_us":573158,"thread_start_us":91,"threads_started":1}]]}
I20260804 14:52:56.974812 29759 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:56.974982 29759 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:56.975592 29759 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:52:56.976199 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:52:56.976262 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:56.976415 29766 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=32, attempt_no=0}) took 1387 ms. Trace:
I20260804 14:52:56.976473 29766 rpcz_store.cc:276] 0804 14:52:55.588819 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:55.588957 (+ 138us) service_pool.cc:227] Handling call
0804 14:52:56.976405 (+1387448us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:56.377894 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:52:56.378045 (+ 151us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:56.378049 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:52:56.378051 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:56.378174 (+ 123us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:56.378187 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:56.378190 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:56.378243 (+ 53us) tablet.cc:699] Row locks acquired
0804 14:52:56.378245 (+ 2us) write_op.cc:260] PREPARE: finished
0804 14:52:56.378297 (+ 52us) write_op.cc:270] Start()
0804 14:52:56.676476 (+298179us) write_op.cc:276] Timestamp: P: 1785855176378274 usec, L: 0
0804 14:52:56.676481 (+ 5us) op_driver.cc:331] REPLICATION: starting
0804 14:52:56.676739 (+ 258us) log.cc:842] Serialized 354 byte log entry
0804 14:52:56.975875 (+299136us) op_driver.cc:444] REPLICATION: finished
0804 14:52:56.975984 (+ 109us) write_op.cc:301] APPLY: starting
0804 14:52:56.976003 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:56.976038 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:56.976055 (+ 17us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:56.976122 (+ 67us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:56.976152 (+ 30us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:56.976156 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:52:56.976243 (+ 87us) log.cc:842] Serialized 43 byte log entry
0804 14:52:56.976301 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:56.976314 (+ 13us) write_op.cc:454] Released schema lock
0804 14:52:56.976319 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":54,"num_ops":1,"prepare.queue_time_us":788471,"prepare.run_cpu_time_us":853,"prepare.run_wall_time_us":298921,"replication_time_us":299370}]]}
I20260804 14:52:56.976868 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:52:56.977070 29759 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:57.257721 29745 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:57.257869 29745 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:57.258416 29745 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:52:57.258848 29096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:52:57.258976 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:57.260306 29187 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:52:57.351924 29745 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:58.103883 29773 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:58.104079 29773 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:58.104856 29773 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:52:58.105441 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:58.105513 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:52:58.105996 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:52:58.106295 29773 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:58.106671 29780 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=33, attempt_no=0}) took 1153 ms. Trace:
I20260804 14:52:58.106776 29780 rpcz_store.cc:276] 0804 14:52:56.953202 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:56.953294 (+ 92us) service_pool.cc:227] Handling call
0804 14:52:58.106657 (+1153363us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:56.953714 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:52:56.953797 (+ 83us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:56.953800 (+ 3us) write_op.cc:435] Acquired schema lock
0804 14:52:56.953801 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:56.953925 (+ 124us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:56.953981 (+ 56us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:56.953984 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:56.954019 (+ 35us) tablet.cc:699] Row locks acquired
0804 14:52:56.954019 (+ 0us) write_op.cc:260] PREPARE: finished
0804 14:52:56.954050 (+ 31us) write_op.cc:270] Start()
0804 14:52:57.665191 (+711141us) write_op.cc:276] Timestamp: P: 1785855176954029 usec, L: 0
0804 14:52:57.665194 (+ 3us) op_driver.cc:331] REPLICATION: starting
0804 14:52:57.665378 (+ 184us) log.cc:842] Serialized 354 byte log entry
0804 14:52:58.105081 (+439703us) op_driver.cc:444] REPLICATION: finished
0804 14:52:58.105378 (+ 297us) write_op.cc:301] APPLY: starting
0804 14:52:58.105398 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:58.105439 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:58.105455 (+ 16us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:58.105522 (+ 67us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:58.106119 (+ 597us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:58.106130 (+ 11us) write_op.cc:312] APPLY: finished
0804 14:52:58.106223 (+ 93us) log.cc:842] Serialized 22 byte log entry
0804 14:52:58.106578 (+ 355us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:58.106589 (+ 11us) write_op.cc:454] Released schema lock
0804 14:52:58.106594 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":240,"num_ops":1,"prepare.queue_time_us":61,"prepare.run_cpu_time_us":683,"prepare.run_wall_time_us":711731,"replication_time_us":439860,"thread_start_us":107,"threads_started":1}]]}
I20260804 14:52:58.421341 29745 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:58.421535 29745 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:58.422128 29745 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:52:58.422505 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:58.422657 29096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:52:58.423132 29185 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:52:58.717345 29773 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:58.717545 29773 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:58.718189 29773 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:52:58.718713 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:58.718791 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:52:58.719144 29792 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=34, attempt_no=0}) took 1741 ms. Trace:
I20260804 14:52:58.719296 29792 rpcz_store.cc:276] 0804 14:52:56.978067 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:56.978134 (+ 67us) service_pool.cc:227] Handling call
0804 14:52:58.719128 (+1740994us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:57.665483 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:52:57.665595 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:57.665599 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:52:57.665601 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:57.665673 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:57.665682 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:57.665685 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:57.665730 (+ 45us) tablet.cc:699] Row locks acquired
0804 14:52:57.665731 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:57.665774 (+ 43us) write_op.cc:270] Start()
0804 14:52:58.209947 (+544173us) write_op.cc:276] Timestamp: P: 1785855177665715 usec, L: 0
0804 14:52:58.209952 (+ 5us) op_driver.cc:331] REPLICATION: starting
0804 14:52:58.210173 (+ 221us) log.cc:842] Serialized 354 byte log entry
0804 14:52:58.718385 (+508212us) op_driver.cc:444] REPLICATION: finished
0804 14:52:58.718702 (+ 317us) write_op.cc:301] APPLY: starting
0804 14:52:58.718720 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0804 14:52:58.718769 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0804 14:52:58.718784 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:52:58.718860 (+ 76us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:52:58.718893 (+ 33us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:52:58.718898 (+ 5us) write_op.cc:312] APPLY: finished
0804 14:52:58.718970 (+ 72us) log.cc:842] Serialized 22 byte log entry
0804 14:52:58.719034 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:52:58.719047 (+ 13us) write_op.cc:454] Released schema lock
0804 14:52:58.719052 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":258,"num_ops":1,"prepare.queue_time_us":686989,"prepare.run_cpu_time_us":683,"prepare.run_wall_time_us":544769,"replication_time_us":508411,"thread_start_us":162,"threads_started":1}]]}
I20260804 14:52:58.719491 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:52:58.913833 29745 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:59.372377 29773 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:52:59.698822 29745 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:52:59.698972 29745 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:52:59.699602 29745 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:52:59.700115 29096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:52:59.700330 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:52:59.700690 29187 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:52:59.700933 29745 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:53:00.277240 29773 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:53:00.277392 29773 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:53:00.277968 29773 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:53:00.278383 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:53:00.278533 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:53:00.278940 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:53:00.428941 29745 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:53:00.429111 29745 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:53:00.429678 29745 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:53:00.430065 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:53:00.430105 29096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:53:00.430583 29187 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:53:00.430788 29745 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:53:00.431083 29807 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=37, attempt_no=0}) took 1514 ms. Trace:
I20260804 14:53:00.431185 29807 rpcz_store.cc:276] 0804 14:52:58.917036 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:58.917104 (+ 68us) service_pool.cc:227] Handling call
0804 14:53:00.431056 (+1513952us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:59.219144 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:52:59.219303 (+ 159us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:59.219307 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:52:59.219309 (+ 2us) tablet.cc:660] Decoding operations
0804 14:52:59.219377 (+ 68us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:59.219386 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:59.219388 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:59.219425 (+ 37us) tablet.cc:699] Row locks acquired
0804 14:52:59.219426 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:52:59.219473 (+ 47us) write_op.cc:270] Start()
0804 14:52:59.934641 (+715168us) write_op.cc:276] Timestamp: P: 1785855179219405 usec, L: 0
0804 14:52:59.934644 (+ 3us) op_driver.cc:331] REPLICATION: starting
0804 14:52:59.934822 (+ 178us) log.cc:842] Serialized 354 byte log entry
0804 14:53:00.430095 (+495273us) op_driver.cc:444] REPLICATION: finished
0804 14:53:00.430345 (+ 250us) write_op.cc:301] APPLY: starting
0804 14:53:00.430613 (+ 268us) tablet.cc:1365] starting BulkCheckPresence
0804 14:53:00.430650 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0804 14:53:00.430671 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:53:00.430737 (+ 66us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:53:00.430765 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:53:00.430769 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:53:00.430879 (+ 110us) log.cc:842] Serialized 43 byte log entry
0804 14:53:00.430940 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:53:00.430955 (+ 15us) write_op.cc:454] Released schema lock
0804 14:53:00.430961 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":3456,"child_traces":[["op",{"apply.queue_time_us":199,"num_ops":1,"prepare.queue_time_us":301544,"prepare.run_cpu_time_us":661,"prepare.run_wall_time_us":715736,"replication_time_us":495428,"thread_start_us":105,"threads_started":1}]]}
I20260804 14:53:00.935041 29745 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:53:00.935227 29745 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:53:00.935792 29745 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:53:00.936162 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:53:00.936193 29096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:53:00.936647 29185 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:53:00.936843 29745 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:53:00.938215 29816 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=38, attempt_no=0}) took 1234 ms. Trace:
I20260804 14:53:00.938309 29816 rpcz_store.cc:276] 0804 14:52:59.702820 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:52:59.702867 (+ 47us) service_pool.cc:227] Handling call
0804 14:53:00.937517 (+1234650us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:52:59.935025 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:52:59.935174 (+ 149us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:52:59.935180 (+ 6us) write_op.cc:435] Acquired schema lock
0804 14:52:59.935181 (+ 1us) tablet.cc:660] Decoding operations
0804 14:52:59.935297 (+ 116us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:52:59.935309 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0804 14:52:59.935313 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:52:59.935361 (+ 48us) tablet.cc:699] Row locks acquired
0804 14:52:59.935363 (+ 2us) write_op.cc:260] PREPARE: finished
0804 14:52:59.935408 (+ 45us) write_op.cc:270] Start()
0804 14:53:00.202589 (+267181us) write_op.cc:276] Timestamp: P: 1785855179935345 usec, L: 0
0804 14:53:00.202594 (+ 5us) op_driver.cc:331] REPLICATION: starting
0804 14:53:00.202816 (+ 222us) log.cc:842] Serialized 354 byte log entry
0804 14:53:00.936836 (+734020us) op_driver.cc:444] REPLICATION: finished
0804 14:53:00.937054 (+ 218us) write_op.cc:301] APPLY: starting
0804 14:53:00.937076 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0804 14:53:00.937114 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
0804 14:53:00.937134 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:53:00.937201 (+ 67us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:53:00.937230 (+ 29us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:53:00.937239 (+ 9us) write_op.cc:312] APPLY: finished
0804 14:53:00.937330 (+ 91us) log.cc:842] Serialized 43 byte log entry
0804 14:53:00.937397 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:53:00.937412 (+ 15us) write_op.cc:454] Released schema lock
0804 14:53:00.937418 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":141,"num_ops":1,"prepare.queue_time_us":231698,"prepare.run_cpu_time_us":774,"prepare.run_wall_time_us":267857,"replication_time_us":734218,"thread_start_us":84,"threads_started":1}]]}
I20260804 14:53:01.458045 29773 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:53:01.458607 29097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.24.6.131:57043 (request call id 7422) took 1180 ms. Trace:
I20260804 14:53:01.458719 29097 rpcz_store.cc:276] 0804 14:53:00.278523 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:53:00.278649 (+ 126us) service_pool.cc:227] Handling call
0804 14:53:00.278837 (+ 188us) raft_consensus.cc:1457] Updating replica for 2 ops
0804 14:53:00.278940 (+ 103us) raft_consensus.cc:1520] Early marking committed up to index 38
0804 14:53:00.279119 (+ 179us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0804 14:53:00.279251 (+ 132us) write_op.cc:270] Start()
0804 14:53:01.161556 (+882305us) write_op.cc:276] Timestamp: P: 1785855179219405 usec, L: 0
0804 14:53:01.163018 (+ 1462us) write_op.cc:270] Start()
0804 14:53:01.457316 (+294298us) write_op.cc:276] Timestamp: P: 1785855179935345 usec, L: 0
0804 14:53:01.457736 (+ 420us) log.cc:842] Serialized 708 byte log entry
0804 14:53:01.457878 (+ 142us) raft_consensus.cc:1631] Marking committed up to 38
0804 14:53:01.457883 (+ 5us) raft_consensus.cc:1683] Filling consensus response to leader.
0804 14:53:01.457945 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0804 14:53:01.458525 (+ 580us) raft_consensus.cc:1671] finished
0804 14:53:01.458527 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0804 14:53:01.458586 (+ 59us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:53:01.162090 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:53:01.162297 (+ 207us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:53:01.162305 (+ 8us) write_op.cc:435] Acquired schema lock
0804 14:53:01.162308 (+ 3us) tablet.cc:660] Decoding operations
0804 14:53:01.162415 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:53:01.162430 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0804 14:53:01.162435 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:53:01.162492 (+ 57us) tablet.cc:699] Row locks acquired
0804 14:53:01.162494 (+ 2us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0804 14:53:01.457673 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:53:01.457785 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:53:01.457789 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:53:01.457791 (+ 2us) tablet.cc:660] Decoding operations
0804 14:53:01.458001 (+ 210us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:53:01.458052 (+ 51us) write_op.cc:641] Partition lock acquired for write op
0804 14:53:01.458055 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:53:01.458126 (+ 71us) tablet.cc:699] Row locks acquired
0804 14:53:01.458127 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":90,"threads_started":1,"wal-append.queue_time_us":296,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":8,"prepare.queue_time_us":308,"prepare.run_cpu_time_us":583,"prepare.run_wall_time_us":630,"thread_start_us":129,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":40,"prepare.queue_time_us":160,"prepare.run_cpu_time_us":571,"prepare.run_wall_time_us":572}]]}
I20260804 14:53:01.544883 29745 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:53:01.545059 29745 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:53:01.545696 29745 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:53:01.546391 29096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:53:01.547079 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:53:01.547744 29185 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:53:01.548149 29745 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:53:01.548197 29829 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=39, attempt_no=0}) took 1115 ms. Trace:
I20260804 14:53:01.548295 29829 rpcz_store.cc:276] 0804 14:53:00.432671 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:53:00.432738 (+ 67us) service_pool.cc:227] Handling call
0804 14:53:01.548179 (+1115441us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:53:00.433112 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:53:00.433190 (+ 78us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:53:00.433193 (+ 3us) write_op.cc:435] Acquired schema lock
0804 14:53:00.433194 (+ 1us) tablet.cc:660] Decoding operations
0804 14:53:00.433254 (+ 60us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:53:00.433260 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0804 14:53:00.433262 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:53:00.433292 (+ 30us) tablet.cc:699] Row locks acquired
0804 14:53:00.433293 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:53:00.433321 (+ 28us) write_op.cc:270] Start()
0804 14:53:00.634518 (+201197us) write_op.cc:276] Timestamp: P: 1785855180433256 usec, L: 0
0804 14:53:00.634523 (+ 5us) op_driver.cc:331] REPLICATION: starting
0804 14:53:00.634776 (+ 253us) log.cc:842] Serialized 354 byte log entry
0804 14:53:01.547306 (+912530us) op_driver.cc:444] REPLICATION: finished
0804 14:53:01.547538 (+ 232us) write_op.cc:301] APPLY: starting
0804 14:53:01.547558 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0804 14:53:01.547606 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0804 14:53:01.547622 (+ 16us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:53:01.547719 (+ 97us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:53:01.547755 (+ 36us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:53:01.547760 (+ 5us) write_op.cc:312] APPLY: finished
0804 14:53:01.547835 (+ 75us) log.cc:842] Serialized 22 byte log entry
0804 14:53:01.548080 (+ 245us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:53:01.548096 (+ 16us) write_op.cc:454] Released schema lock
0804 14:53:01.548102 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":172,"num_ops":1,"prepare.queue_time_us":44,"prepare.run_cpu_time_us":655,"prepare.run_wall_time_us":201736,"replication_time_us":912756,"thread_start_us":102,"threads_started":1}]]}
I20260804 14:53:02.351914 29773 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:53:02.352146 29773 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:53:02.352792 29773 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:53:02.353132 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:53:02.353276 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:53:02.353791 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:53:02.404294 29745 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:53:02.404455 29745 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:53:02.405010 29745 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:53:02.405439 29096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:53:02.405577 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:53:02.406334 29185 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:53:02.406756 29745 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:53:02.407460 29838 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=40, attempt_no=0}) took 1467 ms. Trace:
I20260804 14:53:02.407594 29838 rpcz_store.cc:276] 0804 14:53:00.939714 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:53:00.939789 (+ 75us) service_pool.cc:227] Handling call
0804 14:53:02.407425 (+1467636us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:53:00.940203 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:53:00.940287 (+ 84us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:53:00.940290 (+ 3us) write_op.cc:435] Acquired schema lock
0804 14:53:00.940291 (+ 1us) tablet.cc:660] Decoding operations
0804 14:53:00.940352 (+ 61us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:53:00.940360 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0804 14:53:00.940362 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:53:00.940393 (+ 31us) tablet.cc:699] Row locks acquired
0804 14:53:00.940394 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:53:00.940435 (+ 41us) write_op.cc:270] Start()
0804 14:53:01.829778 (+889343us) write_op.cc:276] Timestamp: P: 1785855180940401 usec, L: 0
0804 14:53:01.829784 (+ 6us) op_driver.cc:331] REPLICATION: starting
0804 14:53:01.830101 (+ 317us) log.cc:842] Serialized 354 byte log entry
0804 14:53:02.406563 (+576462us) op_driver.cc:444] REPLICATION: finished
0804 14:53:02.406781 (+ 218us) write_op.cc:301] APPLY: starting
0804 14:53:02.406802 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0804 14:53:02.406971 (+ 169us) tablet.cc:1368] finished BulkCheckPresence
0804 14:53:02.406987 (+ 16us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:53:02.407056 (+ 69us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:53:02.407087 (+ 31us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:53:02.407091 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:53:02.407224 (+ 133us) log.cc:842] Serialized 43 byte log entry
0804 14:53:02.407297 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:53:02.407312 (+ 15us) write_op.cc:454] Released schema lock
0804 14:53:02.407319 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":165,"num_ops":1,"prepare.queue_time_us":56,"prepare.run_cpu_time_us":1219,"prepare.run_wall_time_us":889977,"replication_time_us":576755,"thread_start_us":77,"threads_started":1}]]}
I20260804 14:53:02.431990 29773 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:53:02.704756 29745 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:53:02.704924 29745 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:53:02.705411 29745 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:53:02.705825 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:53:02.706220 29097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:53:02.706737 29187 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:53:02.706931 29745 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:53:02.857235 29745 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:53:02.857424 29745 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:53:02.858070 29745 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:53:02.858513 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:53:02.858835 29096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:53:02.859043 29838 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=41, attempt_no=0}) took 1309 ms. Trace:
I20260804 14:53:02.859316 29185 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:53:02.859134 29838 rpcz_store.cc:276] 0804 14:53:01.549925 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:53:01.550058 (+ 133us) service_pool.cc:227] Handling call
0804 14:53:02.859025 (+1308967us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:53:01.830206 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:53:01.830376 (+ 170us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:53:01.830384 (+ 8us) write_op.cc:435] Acquired schema lock
0804 14:53:01.830388 (+ 4us) tablet.cc:660] Decoding operations
0804 14:53:01.830474 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:53:01.830483 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0804 14:53:01.830485 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:53:01.830525 (+ 40us) tablet.cc:699] Row locks acquired
0804 14:53:01.830526 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:53:01.830566 (+ 40us) write_op.cc:270] Start()
0804 14:53:02.739698 (+909132us) write_op.cc:276] Timestamp: P: 1785855181830520 usec, L: 0
0804 14:53:02.739702 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:53:02.739878 (+ 176us) log.cc:842] Serialized 354 byte log entry
0804 14:53:02.858477 (+118599us) op_driver.cc:444] REPLICATION: finished
0804 14:53:02.858589 (+ 112us) write_op.cc:301] APPLY: starting
0804 14:53:02.858608 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0804 14:53:02.858640 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0804 14:53:02.858654 (+ 14us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:53:02.858712 (+ 58us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:53:02.858740 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:53:02.858747 (+ 7us) write_op.cc:312] APPLY: finished
0804 14:53:02.858845 (+ 98us) log.cc:842] Serialized 43 byte log entry
0804 14:53:02.858903 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:53:02.858917 (+ 14us) write_op.cc:454] Released schema lock
0804 14:53:02.858922 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":70,"num_ops":1,"prepare.queue_time_us":279718,"prepare.run_cpu_time_us":750,"prepare.run_wall_time_us":909804,"raft.queue_time_us":307,"raft.run_cpu_time_us":363,"raft.run_wall_time_us":364,"replication_time_us":118753,"spinlock_wait_cycles":20736}]]}
I20260804 14:53:02.859758 29745 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:53:03.221678 29773 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:53:03.221817 29773 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:53:03.222348 29773 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:53:03.222810 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:53:03.222954 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:53:03.223513 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:53:03.439913 29745 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:53:03.440099 29745 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:53:03.440754 29745 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:53:03.441201 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 44 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:53:03.441629 29096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 44 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:53:03.442168 29846 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=42, attempt_no=0}) took 1032 ms. Trace:
I20260804 14:53:03.442309 29185 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:53:03.442492 29745 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:53:03.442307 29846 rpcz_store.cc:276] 0804 14:53:02.409318 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:53:02.409368 (+ 50us) service_pool.cc:227] Handling call
0804 14:53:03.442148 (+1032780us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:53:02.740048 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:53:02.740164 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:53:02.740168 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:53:02.740170 (+ 2us) tablet.cc:660] Decoding operations
0804 14:53:02.740261 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:53:02.740272 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0804 14:53:02.740275 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:53:02.740324 (+ 49us) tablet.cc:699] Row locks acquired
0804 14:53:02.740326 (+ 2us) write_op.cc:260] PREPARE: finished
0804 14:53:02.740374 (+ 48us) write_op.cc:270] Start()
0804 14:53:02.869632 (+129258us) write_op.cc:276] Timestamp: P: 1785855182740340 usec, L: 0
0804 14:53:02.869636 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:53:02.869812 (+ 176us) log.cc:842] Serialized 354 byte log entry
0804 14:53:03.441447 (+571635us) op_driver.cc:444] REPLICATION: finished
0804 14:53:03.441693 (+ 246us) write_op.cc:301] APPLY: starting
0804 14:53:03.441711 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0804 14:53:03.441745 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0804 14:53:03.441759 (+ 14us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:53:03.441824 (+ 65us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:53:03.441852 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:53:03.441861 (+ 9us) write_op.cc:312] APPLY: finished
0804 14:53:03.441959 (+ 98us) log.cc:842] Serialized 43 byte log entry
0804 14:53:03.442021 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:53:03.442036 (+ 15us) write_op.cc:454] Released schema lock
0804 14:53:03.442043 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":192,"num_ops":1,"prepare.queue_time_us":330265,"prepare.run_cpu_time_us":644,"prepare.run_wall_time_us":129820,"replication_time_us":571627,"thread_start_us":73,"threads_started":1}]]}
I20260804 14:53:03.921756 29773 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:53:04.005070 29745 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:53:04.005262 29745 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:53:04.005854 29745 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:53:04.006461 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:53:04.006532 29096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:53:04.007074 29185 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:53:04.007308 29745 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:53:04.007814 29853 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=43, attempt_no=0}) took 1147 ms. Trace:
I20260804 14:53:04.007925 29853 rpcz_store.cc:276] 0804 14:53:02.860732 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:53:02.860801 (+ 69us) service_pool.cc:227] Handling call
0804 14:53:04.007794 (+1146993us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:53:02.869891 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:53:02.870035 (+ 144us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:53:02.870040 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:53:02.870042 (+ 2us) tablet.cc:660] Decoding operations
0804 14:53:02.870134 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:53:02.870147 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0804 14:53:02.870151 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:53:02.870195 (+ 44us) tablet.cc:699] Row locks acquired
0804 14:53:02.870197 (+ 2us) write_op.cc:260] PREPARE: finished
0804 14:53:02.870240 (+ 43us) write_op.cc:270] Start()
0804 14:53:03.776429 (+906189us) write_op.cc:276] Timestamp: P: 1785855182870195 usec, L: 0
0804 14:53:03.776434 (+ 5us) op_driver.cc:331] REPLICATION: starting
0804 14:53:03.776672 (+ 238us) log.cc:842] Serialized 354 byte log entry
0804 14:53:04.006911 (+230239us) op_driver.cc:444] REPLICATION: finished
0804 14:53:04.007156 (+ 245us) write_op.cc:301] APPLY: starting
0804 14:53:04.007175 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0804 14:53:04.007352 (+ 177us) tablet.cc:1368] finished BulkCheckPresence
0804 14:53:04.007373 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:53:04.007452 (+ 79us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:53:04.007489 (+ 37us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:53:04.007493 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:53:04.007584 (+ 91us) log.cc:842] Serialized 43 byte log entry
0804 14:53:04.007671 (+ 87us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:53:04.007687 (+ 16us) write_op.cc:454] Released schema lock
0804 14:53:04.007694 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":177,"num_ops":1,"prepare.queue_time_us":8725,"prepare.run_cpu_time_us":800,"prepare.run_wall_time_us":906849,"replication_time_us":230452,"thread_start_us":96,"threads_started":1}]]}
I20260804 14:53:04.060542 29773 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:53:04.060724 29773 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:53:04.061290 29773 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:53:04.061769 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:53:04.061795 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:53:04.062244 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:53:04.062677 29773 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:53:04.766522 29773 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:53:04.766693 29773 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:53:04.767246 29773 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:53:04.767880 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:53:04.767902 29272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:53:04.768467 29861 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=44, attempt_no=0}) took 1324 ms. Trace:
I20260804 14:53:04.768507 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:53:04.768682 29773 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:53:04.768597 29861 rpcz_store.cc:276] 0804 14:53:03.444171 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:53:03.444236 (+ 65us) service_pool.cc:227] Handling call
0804 14:53:04.768436 (+1324200us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:53:03.776789 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:53:03.776931 (+ 142us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:53:03.776935 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:53:03.776937 (+ 2us) tablet.cc:660] Decoding operations
0804 14:53:03.777008 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:53:03.777018 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0804 14:53:03.777020 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:53:03.777058 (+ 38us) tablet.cc:699] Row locks acquired
0804 14:53:03.777059 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:53:03.777098 (+ 39us) write_op.cc:270] Start()
0804 14:53:04.520234 (+743136us) write_op.cc:276] Timestamp: P: 1785855183777052 usec, L: 0
0804 14:53:04.520238 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:53:04.520461 (+ 223us) log.cc:842] Serialized 354 byte log entry
0804 14:53:04.767591 (+247130us) op_driver.cc:444] REPLICATION: finished
0804 14:53:04.767819 (+ 228us) write_op.cc:301] APPLY: starting
0804 14:53:04.767837 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0804 14:53:04.767869 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0804 14:53:04.767882 (+ 13us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:53:04.767945 (+ 63us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:53:04.767971 (+ 26us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:53:04.767975 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:53:04.768087 (+ 112us) log.cc:842] Serialized 43 byte log entry
0804 14:53:04.768309 (+ 222us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:53:04.768323 (+ 14us) write_op.cc:454] Released schema lock
0804 14:53:04.768328 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":178,"num_ops":1,"prepare.queue_time_us":332140,"prepare.run_cpu_time_us":680,"prepare.run_wall_time_us":743737,"replication_time_us":247329,"thread_start_us":83,"threads_started":1}]]}
I20260804 14:53:04.952512 29857 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:53:04.952678 29857 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:53:04.953276 29857 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:53:04.953614 29097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:53:04.953974 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:53:04.954437 29187 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:53:04.954669 29857 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:53:05.524230 29773 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:53:05.524405 29773 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:53:05.524951 29773 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:53:05.525427 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:53:05.525465 29272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:53:05.526003 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:53:05.526229 29773 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:53:05.526803 29869 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=45, attempt_no=0}) took 1517 ms. Trace:
I20260804 14:53:05.526872 29869 rpcz_store.cc:276] 0804 14:53:04.009531 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:53:04.009593 (+ 62us) service_pool.cc:227] Handling call
0804 14:53:05.526767 (+1517174us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:53:04.520564 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:53:04.520674 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:53:04.520679 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:53:04.520680 (+ 1us) tablet.cc:660] Decoding operations
0804 14:53:04.520762 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:53:04.520772 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0804 14:53:04.520775 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:53:04.520826 (+ 51us) tablet.cc:699] Row locks acquired
0804 14:53:04.520827 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:53:04.520872 (+ 45us) write_op.cc:270] Start()
0804 14:53:05.259089 (+738217us) write_op.cc:276] Timestamp: P: 1785855184520809 usec, L: 0
0804 14:53:05.259095 (+ 6us) op_driver.cc:331] REPLICATION: starting
0804 14:53:05.259324 (+ 229us) log.cc:842] Serialized 354 byte log entry
0804 14:53:05.525390 (+266066us) op_driver.cc:444] REPLICATION: finished
0804 14:53:05.525678 (+ 288us) write_op.cc:301] APPLY: starting
0804 14:53:05.525696 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0804 14:53:05.525859 (+ 163us) tablet.cc:1368] finished BulkCheckPresence
0804 14:53:05.525877 (+ 18us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:53:05.525945 (+ 68us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:53:05.525974 (+ 29us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:53:05.525978 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:53:05.526065 (+ 87us) log.cc:842] Serialized 43 byte log entry
0804 14:53:05.526123 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:53:05.526137 (+ 14us) write_op.cc:454] Released schema lock
0804 14:53:05.526142 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":233,"num_ops":1,"prepare.queue_time_us":510457,"prepare.run_cpu_time_us":855,"prepare.run_wall_time_us":738913,"raft.queue_time_us":167,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":266269,"spinlock_wait_cycles":113536,"thread_start_us":93,"threads_started":1}]]}
I20260804 14:53:05.756460 29773 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:53:05.756630 29773 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:53:05.757261 29773 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:53:05.757951 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:53:05.757956 29272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:53:05.758452 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:53:05.758615 29773 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:53:06.095405 29857 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:53:06.095595 29857 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:53:06.096163 29857 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:53:06.096621 29096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:53:06.096661 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:53:06.097288 29187 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:53:06.562170 29773 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:53:06.562335 29773 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:53:06.562991 29773 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:53:06.563611 29272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 49 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:53:06.563783 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 49 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:53:06.564254 29877 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36888 (ReqId={client: 2841f758dbb24ed8842b216d549cd17b, seq_no=47, attempt_no=0}) took 1035 ms. Trace:
I20260804 14:53:06.564378 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:53:06.564371 29877 rpcz_store.cc:276] 0804 14:53:05.528432 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:53:05.528517 (+ 85us) service_pool.cc:227] Handling call
0804 14:53:06.564237 (+1035720us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:53:05.529030 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:53:05.529140 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:53:05.529143 (+ 3us) write_op.cc:435] Acquired schema lock
0804 14:53:05.529145 (+ 2us) tablet.cc:660] Decoding operations
0804 14:53:05.529206 (+ 61us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:53:05.529213 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0804 14:53:05.529215 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:53:05.529245 (+ 30us) tablet.cc:699] Row locks acquired
0804 14:53:05.529246 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:53:05.529275 (+ 29us) write_op.cc:270] Start()
0804 14:53:05.773435 (+244160us) write_op.cc:276] Timestamp: P: 1785855185529226 usec, L: 0
0804 14:53:05.773439 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:53:05.773616 (+ 177us) log.cc:842] Serialized 354 byte log entry
0804 14:53:06.563357 (+789741us) op_driver.cc:444] REPLICATION: finished
0804 14:53:06.563645 (+ 288us) write_op.cc:301] APPLY: starting
0804 14:53:06.563663 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0804 14:53:06.563697 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0804 14:53:06.563710 (+ 13us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:53:06.563771 (+ 61us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:53:06.563803 (+ 32us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:53:06.563808 (+ 5us) write_op.cc:312] APPLY: finished
0804 14:53:06.563888 (+ 80us) log.cc:842] Serialized 43 byte log entry
0804 14:53:06.564120 (+ 232us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:53:06.564133 (+ 13us) write_op.cc:454] Released schema lock
0804 14:53:06.564140 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":240,"num_ops":1,"prepare.queue_time_us":141,"prepare.run_cpu_time_us":593,"prepare.run_wall_time_us":244682,"replication_time_us":789892,"thread_start_us":86,"threads_started":1}]]}
I20260804 14:53:06.653087 29773 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:53:06.735123 29420 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260804 14:53:06.735646 29066 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260804 14:53:07.124720 29857 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:53:07.125103 29275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.24.6.131:48343 (request call id 4721) took 1028 ms. Trace:
I20260804 14:53:07.125188 29275 rpcz_store.cc:276] 0804 14:53:06.096856 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:53:06.097034 (+ 178us) service_pool.cc:227] Handling call
0804 14:53:06.097207 (+ 173us) raft_consensus.cc:1457] Updating replica for 3 ops
0804 14:53:06.097295 (+ 88us) raft_consensus.cc:1520] Early marking committed up to index 47
0804 14:53:06.097479 (+ 184us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0804 14:53:06.097595 (+ 116us) write_op.cc:270] Start()
0804 14:53:06.562764 (+465169us) write_op.cc:276] Timestamp: P: 1785855185259818 usec, L: 0
0804 14:53:06.563036 (+ 272us) write_op.cc:270] Start()
0804 14:53:07.121200 (+558164us) write_op.cc:276] Timestamp: P: 1785855185529226 usec, L: 0
0804 14:53:07.121585 (+ 385us) write_op.cc:270] Start()
0804 14:53:07.123796 (+ 2211us) write_op.cc:276] Timestamp: P: 1785855185774067 usec, L: 0
0804 14:53:07.124282 (+ 486us) log.cc:842] Serialized 1062 byte log entry
0804 14:53:07.124453 (+ 171us) raft_consensus.cc:1631] Marking committed up to 48
0804 14:53:07.124625 (+ 172us) raft_consensus.cc:1683] Filling consensus response to leader.
0804 14:53:07.124662 (+ 37us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0804 14:53:07.125024 (+ 362us) raft_consensus.cc:1671] finished
0804 14:53:07.125026 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0804 14:53:07.125085 (+ 59us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:53:06.562963 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:53:06.563059 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:53:06.563063 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:53:06.563065 (+ 2us) tablet.cc:660] Decoding operations
0804 14:53:06.563133 (+ 68us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:53:06.563150 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0804 14:53:06.563156 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:53:06.563224 (+ 68us) tablet.cc:699] Row locks acquired
0804 14:53:06.563225 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:53:07.124485 (+561260us) op_driver.cc:444] REPLICATION: finished
0804 14:53:07.124716 (+ 231us) write_op.cc:301] APPLY: starting
0804 14:53:07.124732 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0804 14:53:07.124767 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0804 14:53:07.124771 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:53:07.124835 (+ 64us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:53:07.124861 (+ 26us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:53:07.124882 (+ 21us) write_op.cc:312] APPLY: finished
0804 14:53:07.124971 (+ 89us) log.cc:842] Serialized 43 byte log entry
0804 14:53:07.125046 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:53:07.125058 (+ 12us) write_op.cc:454] Released schema lock
0804 14:53:07.125063 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0804 14:53:07.125154 (+ 91us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:53:07.125157 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0804 14:53:07.121516 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:53:07.121637 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:53:07.121642 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:53:07.121645 (+ 3us) tablet.cc:660] Decoding operations
0804 14:53:07.121759 (+ 114us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:53:07.121769 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0804 14:53:07.121771 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:53:07.121810 (+ 39us) tablet.cc:699] Row locks acquired
0804 14:53:07.121811 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0804 14:53:07.124135 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0118633f5af24844b38a60678228343e
0804 14:53:07.124227 (+ 92us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:53:07.124231 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:53:07.124232 (+ 1us) tablet.cc:660] Decoding operations
0804 14:53:07.124297 (+ 65us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:53:07.124308 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0804 14:53:07.124310 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:53:07.124344 (+ 34us) tablet.cc:699] Row locks acquired
0804 14:53:07.124345 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":107,"threads_started":1,"wal-append.queue_time_us":222,"child_traces":[["op",{"apply.queue_time_us":177,"apply.run_cpu_time_us":488,"apply.run_wall_time_us":487,"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":51,"prepare.run_cpu_time_us":361,"prepare.run_wall_time_us":362,"replication_time_us":1026875,"thread_start_us":79,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":203,"prepare.run_cpu_time_us":377,"prepare.run_wall_time_us":377,"thread_start_us":111,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":4,"prepare.queue_time_us":194,"prepare.run_cpu_time_us":273,"prepare.run_wall_time_us":273}]]}
I20260804 14:53:07.127581 29252 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
d48d86cb47d943c8b62c82944326ff2d | 127.24.6.190:39627 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.24.6.148:34763 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
02bbbb8aa9ba40d492afb93419678550 | 127.24.6.129:41707 | HEALTHY | <none> | 0 | 0
30a15be32d8d43a6a4f432dd9b6a6869 | 127.24.6.131:38993 | HEALTHY | <none> | 1 | 0
a310b164338b437bb1577d62a34897a0 | 127.24.6.130:33743 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
inject_latency_ms_before_starting_op | 1000 | hidden,advanced | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.24.6.129 | experimental | 127.24.6.129:41707
local_ip_for_outbound_sockets | 127.24.6.130 | experimental | 127.24.6.130:33743
local_ip_for_outbound_sockets | 127.24.6.131 | experimental | 127.24.6.131:38993
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.24.6.129:41707
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.24.6.130:33743
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.24.6.131:38993
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.24.6.148:34763 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260804 14:53:07.157706 24602 log_verifier.cc:126] Checking tablet 0118633f5af24844b38a60678228343e
I20260804 14:53:07.176566 29857 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:53:07.176730 29857 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:53:07.177295 29857 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:53:07.178311 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:53:07.178092 29096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:53:07.179253 29185 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:53:07.179574 29857 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:53:07.191361 29928 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:53:07.191529 29928 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:53:07.192111 29928 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), a310b164338b437bb1577d62a34897a0 (127.24.6.130:33743)
I20260804 14:53:07.192466 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
I20260804 14:53:07.192489 29275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "02bbbb8aa9ba40d492afb93419678550" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "a310b164338b437bb1577d62a34897a0" is_pre_election: true
I20260804 14:53:07.193212 29006 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02bbbb8aa9ba40d492afb93419678550; no voters: 30a15be32d8d43a6a4f432dd9b6a6869, a310b164338b437bb1577d62a34897a0
I20260804 14:53:07.193382 29928 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P 02bbbb8aa9ba40d492afb93419678550 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:53:07.207710 24602 log_verifier.cc:177] Verified matching terms for 50 ops in tablet 0118633f5af24844b38a60678228343e
I20260804 14:53:07.226462 29857 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:53:07.226626 29857 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:53:07.227092 29857 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:53:07.227535 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
W20260804 14:53:07.235101 29185 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.24.6.129:41707 timed out after 0.006s (SENT) [suppressed 1 similar messages]
I20260804 14:53:07.236568 29097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "02bbbb8aa9ba40d492afb93419678550" is_pre_election: true
I20260804 14:53:07.237013 29185 leader_election.cc:302] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310b164338b437bb1577d62a34897a0; no voters: 02bbbb8aa9ba40d492afb93419678550, 30a15be32d8d43a6a4f432dd9b6a6869
I20260804 14:53:07.238432 29857 raft_consensus.cc:2726] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260804 14:53:07.238746 29096 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.24.6.130:49579 (request call id 34) took 8 ms (client timeout 6 ms). Trace:
W20260804 14:53:07.238845 29096 rpcz_store.cc:269] 0804 14:53:07.229824 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:53:07.238621 (+ 8797us) service_pool.cc:214] Skipping call since client already timed out
0804 14:53:07.238730 (+ 109us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260804 14:53:07.256891 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 28982
I20260804 14:53:07.263427 29857 raft_consensus.cc:482] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30a15be32d8d43a6a4f432dd9b6a6869)
I20260804 14:53:07.263582 29857 raft_consensus.cc:504] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 38993 } } peers { permanent_uuid: "a310b164338b437bb1577d62a34897a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33743 } } peers { permanent_uuid: "02bbbb8aa9ba40d492afb93419678550" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41707 } }
I20260804 14:53:07.264110 29857 leader_election.cc:288] T 0118633f5af24844b38a60678228343e P a310b164338b437bb1577d62a34897a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a15be32d8d43a6a4f432dd9b6a6869 (127.24.6.131:38993), 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707)
I20260804 14:53:07.264591 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118633f5af24844b38a60678228343e" candidate_uuid: "a310b164338b437bb1577d62a34897a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "30a15be32d8d43a6a4f432dd9b6a6869" is_pre_election: true
W20260804 14:53:07.273417 29362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.6.129:41707 (error 108) [suppressed 1 similar messages]
W20260804 14:53:07.273460 29185 connection.cc:535] client connection to 127.24.6.129:41707 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260804 14:53:07.274767 29362 consensus_peers.cc:487] T 0118633f5af24844b38a60678228343e P 30a15be32d8d43a6a4f432dd9b6a6869 -> Peer 02bbbb8aa9ba40d492afb93419678550 (127.24.6.129:41707): Couldn't send request to peer 02bbbb8aa9ba40d492afb93419678550. Status: Network error: Client connection negotiation failed: client connection to 127.24.6.129:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260804 14:53:07.274910 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 29162
W20260804 14:53:07.292131 29364 connection.cc:535] client connection to 127.24.6.130:33743 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260804 14:53:07.292538 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 29342
I20260804 14:53:07.306681 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 28868
2026-08-04T14:53:07Z chronyd exiting
[ OK ] RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys (32326 ms)
[ RUN ] RaftConsensusElectionITest.AutomaticLeaderElection
I20260804 14:53:07.340265 24602 test_util.cc:268] Using random seed: -804271133
I20260804 14:53:07.341166 24602 ts_itest-base.cc:112] Starting cluster with:
I20260804 14:53:07.341208 24602 ts_itest-base.cc:113] --------------
I20260804 14:53:07.341220 24602 ts_itest-base.cc:114] 5 tablet servers
I20260804 14:53:07.341233 24602 ts_itest-base.cc:115] 5 replicas per TS
I20260804 14:53:07.341243 24602 ts_itest-base.cc:116] --------------
2026-08-04T14:53:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-04T14:53:07Z Disabled control of system clock
I20260804 14:53:07.354370 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.6.190:42521
--webserver_interface=127.24.6.190
--webserver_port=0
--builtin_ntp_servers=127.24.6.148:33757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.6.190:42521 with env {}
W20260804 14:53:07.477555 29960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:07.477933 29960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:07.478015 29960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:07.482504 29960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260804 14:53:07.482625 29960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:07.482663 29960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260804 14:53:07.482707 29960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260804 14:53:07.488018 29960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:33757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.6.190:42521
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.6.190:42521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.24.6.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.29960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:07.489328 29960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:07.490885 29960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:07.500725 29971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:07.500799 29968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:07.501075 29966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:07.501237 29960 server_base.cc:1034] running on GCE node
I20260804 14:53:07.501971 29960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:07.503412 29960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:07.504618 29960 hybrid_clock.cc:648] HybridClock initialized: now 1785855187504599 us; error 35 us; skew 500 ppm
I20260804 14:53:07.506997 29960 webserver.cc:460] Webserver started at http://127.24.6.190:34279/ using document root <none> and password file <none>
I20260804 14:53:07.507736 29960 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:07.507813 29960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:07.508076 29960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:07.510262 29960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "75412ca9bcac463f8e54d370101ee5ff"
format_stamp: "Formatted at 2026-08-04 14:53:07 on dist-test-slave-92m1"
I20260804 14:53:07.510921 29960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "75412ca9bcac463f8e54d370101ee5ff"
format_stamp: "Formatted at 2026-08-04 14:53:07 on dist-test-slave-92m1"
I20260804 14:53:07.515313 29960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260804 14:53:07.517987 29980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:07.519440 29960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260804 14:53:07.519594 29960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
uuid: "75412ca9bcac463f8e54d370101ee5ff"
format_stamp: "Formatted at 2026-08-04 14:53:07 on dist-test-slave-92m1"
I20260804 14:53:07.519724 29960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:07.542333 29960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:07.543509 29960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:07.543740 29960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:07.553544 30050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.190:42521 every 8 connection(s)
I20260804 14:53:07.553555 29960 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.190:42521
I20260804 14:53:07.554769 29960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260804 14:53:07.558408 30051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:07.562418 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 29960
I20260804 14:53:07.562538 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260804 14:53:07.564625 30051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75412ca9bcac463f8e54d370101ee5ff: Bootstrap starting.
I20260804 14:53:07.567162 30051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75412ca9bcac463f8e54d370101ee5ff: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:07.568056 30051 log.cc:824] T 00000000000000000000000000000000 P 75412ca9bcac463f8e54d370101ee5ff: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:07.570044 30051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75412ca9bcac463f8e54d370101ee5ff: No bootstrap required, opened a new log
I20260804 14:53:07.574220 30051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75412ca9bcac463f8e54d370101ee5ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75412ca9bcac463f8e54d370101ee5ff" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 42521 } }
I20260804 14:53:07.574486 30051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75412ca9bcac463f8e54d370101ee5ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:53:07.574522 30051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75412ca9bcac463f8e54d370101ee5ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75412ca9bcac463f8e54d370101ee5ff, State: Initialized, Role: FOLLOWER
I20260804 14:53:07.575063 30051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75412ca9bcac463f8e54d370101ee5ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75412ca9bcac463f8e54d370101ee5ff" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 42521 } }
I20260804 14:53:07.575294 30051 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75412ca9bcac463f8e54d370101ee5ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260804 14:53:07.575369 30051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75412ca9bcac463f8e54d370101ee5ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260804 14:53:07.575495 30051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75412ca9bcac463f8e54d370101ee5ff [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:07.576496 30051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75412ca9bcac463f8e54d370101ee5ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75412ca9bcac463f8e54d370101ee5ff" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 42521 } }
I20260804 14:53:07.576896 30051 leader_election.cc:302] T 00000000000000000000000000000000 P 75412ca9bcac463f8e54d370101ee5ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75412ca9bcac463f8e54d370101ee5ff; no voters:
I20260804 14:53:07.577248 30051 leader_election.cc:288] T 00000000000000000000000000000000 P 75412ca9bcac463f8e54d370101ee5ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260804 14:53:07.577431 30061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75412ca9bcac463f8e54d370101ee5ff [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:07.577688 30061 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75412ca9bcac463f8e54d370101ee5ff [term 1 LEADER]: Becoming Leader. State: Replica: 75412ca9bcac463f8e54d370101ee5ff, State: Running, Role: LEADER
I20260804 14:53:07.578018 30061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75412ca9bcac463f8e54d370101ee5ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75412ca9bcac463f8e54d370101ee5ff" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 42521 } }
I20260804 14:53:07.578627 30051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75412ca9bcac463f8e54d370101ee5ff [sys.catalog]: configured and running, proceeding with master startup.
I20260804 14:53:07.579909 30062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75412ca9bcac463f8e54d370101ee5ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75412ca9bcac463f8e54d370101ee5ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75412ca9bcac463f8e54d370101ee5ff" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 42521 } } }
I20260804 14:53:07.579926 30063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75412ca9bcac463f8e54d370101ee5ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75412ca9bcac463f8e54d370101ee5ff. Latest consensus state: current_term: 1 leader_uuid: "75412ca9bcac463f8e54d370101ee5ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75412ca9bcac463f8e54d370101ee5ff" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 42521 } } }
I20260804 14:53:07.580081 30062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75412ca9bcac463f8e54d370101ee5ff [sys.catalog]: This master's current role is: LEADER
I20260804 14:53:07.580106 30063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75412ca9bcac463f8e54d370101ee5ff [sys.catalog]: This master's current role is: LEADER
I20260804 14:53:07.580715 30073 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260804 14:53:07.584637 30073 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260804 14:53:07.593129 30073 catalog_manager.cc:1347] Generated new cluster ID: 04898069f3794841802c9d4272a882a2
I20260804 14:53:07.593254 30073 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260804 14:53:07.600244 30073 catalog_manager.cc:1370] Generated new certificate authority record
I20260804 14:53:07.601598 30073 catalog_manager.cc:1504] Loading token signing keys...
I20260804 14:53:07.618646 30073 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75412ca9bcac463f8e54d370101ee5ff: Generated new TSK 0
I20260804 14:53:07.619773 30073 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260804 14:53:07.626036 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.129:0
--local_ip_for_outbound_sockets=127.24.6.129
--webserver_interface=127.24.6.129
--webserver_port=0
--tserver_master_addrs=127.24.6.190:42521
--builtin_ntp_servers=127.24.6.148:33757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
W20260804 14:53:07.753304 30089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:07.753697 30089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:07.753780 30089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:07.758293 30089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:07.758471 30089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.129
I20260804 14:53:07.763825 30089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:33757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.24.6.129
--webserver_port=0
--tserver_master_addrs=127.24.6.190:42521
--never_fsync=true
--heap_profile_path=/tmp/kudu.30089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.129
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:07.765130 30089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:07.766669 30089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:07.777413 30097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:07.777410 30098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:07.777418 30101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:07.777943 30089 server_base.cc:1034] running on GCE node
I20260804 14:53:07.778473 30089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:07.779366 30089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:07.780580 30089 hybrid_clock.cc:648] HybridClock initialized: now 1785855187780551 us; error 36 us; skew 500 ppm
I20260804 14:53:07.783006 30089 webserver.cc:460] Webserver started at http://127.24.6.129:37977/ using document root <none> and password file <none>
I20260804 14:53:07.783766 30089 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:07.783847 30089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:07.784080 30089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:07.786429 30089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "172e2da08d1b44a78b70acbeb59a0913"
format_stamp: "Formatted at 2026-08-04 14:53:07 on dist-test-slave-92m1"
I20260804 14:53:07.787267 30089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "172e2da08d1b44a78b70acbeb59a0913"
format_stamp: "Formatted at 2026-08-04 14:53:07 on dist-test-slave-92m1"
I20260804 14:53:07.791946 30089 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260804 14:53:07.794901 30108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:07.797359 30089 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260804 14:53:07.797566 30089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "172e2da08d1b44a78b70acbeb59a0913"
format_stamp: "Formatted at 2026-08-04 14:53:07 on dist-test-slave-92m1"
I20260804 14:53:07.797716 30089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:07.811514 30089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:07.812480 30089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:07.812739 30089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:07.813452 30089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:53:07.814553 30089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:53:07.814631 30089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:07.814702 30089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:53:07.814755 30089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:07.824831 30089 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.129:38929
I20260804 14:53:07.824849 30271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.129:38929 every 8 connection(s)
I20260804 14:53:07.825966 30089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260804 14:53:07.835492 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 30089
I20260804 14:53:07.835649 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260804 14:53:07.838290 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.130:0
--local_ip_for_outbound_sockets=127.24.6.130
--webserver_interface=127.24.6.130
--webserver_port=0
--tserver_master_addrs=127.24.6.190:42521
--builtin_ntp_servers=127.24.6.148:33757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20260804 14:53:07.840818 30272 heartbeater.cc:347] Connected to a master server at 127.24.6.190:42521
I20260804 14:53:07.841180 30272 heartbeater.cc:464] Registering TS with master...
I20260804 14:53:07.841890 30272 heartbeater.cc:511] Master 127.24.6.190:42521 requested a full tablet report, sending...
I20260804 14:53:07.843849 29992 ts_manager.cc:194] Registered new tserver with Master: 172e2da08d1b44a78b70acbeb59a0913 (127.24.6.129:38929)
I20260804 14:53:07.845146 29992 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.129:50441
W20260804 14:53:07.963591 30276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:07.963963 30276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:07.964072 30276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:07.968622 30276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:07.968798 30276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.130
I20260804 14:53:07.974038 30276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:33757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.24.6.130
--webserver_port=0
--tserver_master_addrs=127.24.6.190:42521
--never_fsync=true
--heap_profile_path=/tmp/kudu.30276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.130
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:07.975394 30276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:07.977017 30276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:07.988023 30285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:07.988008 30286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:07.988005 30290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:07.988359 30276 server_base.cc:1034] running on GCE node
I20260804 14:53:07.988948 30276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:07.989781 30276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:07.990983 30276 hybrid_clock.cc:648] HybridClock initialized: now 1785855187990956 us; error 44 us; skew 500 ppm
I20260804 14:53:07.993363 30276 webserver.cc:460] Webserver started at http://127.24.6.130:44797/ using document root <none> and password file <none>
I20260804 14:53:07.994040 30276 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:07.994112 30276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:07.994398 30276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:07.996687 30276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "3229081dc1fd49d4a4efd147a9ae069a"
format_stamp: "Formatted at 2026-08-04 14:53:07 on dist-test-slave-92m1"
I20260804 14:53:07.997439 30276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "3229081dc1fd49d4a4efd147a9ae069a"
format_stamp: "Formatted at 2026-08-04 14:53:07 on dist-test-slave-92m1"
I20260804 14:53:08.002208 30276 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260804 14:53:08.005534 30300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:08.007406 30276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260804 14:53:08.007586 30276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "3229081dc1fd49d4a4efd147a9ae069a"
format_stamp: "Formatted at 2026-08-04 14:53:07 on dist-test-slave-92m1"
I20260804 14:53:08.007737 30276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:08.032330 30276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:08.033183 30276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:08.033418 30276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:08.034118 30276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:53:08.035260 30276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:53:08.035346 30276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:08.035426 30276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:53:08.035480 30276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:08.046168 30276 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.130:44109
I20260804 14:53:08.046202 30447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.130:44109 every 8 connection(s)
I20260804 14:53:08.047446 30276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260804 14:53:08.048141 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 30276
I20260804 14:53:08.048318 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260804 14:53:08.051298 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.131:0
--local_ip_for_outbound_sockets=127.24.6.131
--webserver_interface=127.24.6.131
--webserver_port=0
--tserver_master_addrs=127.24.6.190:42521
--builtin_ntp_servers=127.24.6.148:33757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20260804 14:53:08.061602 30451 heartbeater.cc:347] Connected to a master server at 127.24.6.190:42521
I20260804 14:53:08.061954 30451 heartbeater.cc:464] Registering TS with master...
I20260804 14:53:08.062779 30451 heartbeater.cc:511] Master 127.24.6.190:42521 requested a full tablet report, sending...
I20260804 14:53:08.064275 29992 ts_manager.cc:194] Registered new tserver with Master: 3229081dc1fd49d4a4efd147a9ae069a (127.24.6.130:44109)
I20260804 14:53:08.064908 29992 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.130:55459
W20260804 14:53:08.208369 30455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:08.208738 30455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:08.208843 30455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:08.213991 30455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:08.214133 30455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.131
I20260804 14:53:08.219794 30455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:33757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.24.6.131
--webserver_port=0
--tserver_master_addrs=127.24.6.190:42521
--never_fsync=true
--heap_profile_path=/tmp/kudu.30455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.131
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:08.221122 30455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:08.222750 30455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:08.234205 30465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:08.234229 30462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:08.234229 30463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:08.234449 30455 server_base.cc:1034] running on GCE node
I20260804 14:53:08.235152 30455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:08.235999 30455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:08.237275 30455 hybrid_clock.cc:648] HybridClock initialized: now 1785855188237244 us; error 47 us; skew 500 ppm
I20260804 14:53:08.239915 30455 webserver.cc:460] Webserver started at http://127.24.6.131:45485/ using document root <none> and password file <none>
I20260804 14:53:08.240710 30455 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:08.240830 30455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:08.241115 30455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:08.243673 30455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "bcf9b8d5315b4c46b04a2249b3646798"
format_stamp: "Formatted at 2026-08-04 14:53:08 on dist-test-slave-92m1"
I20260804 14:53:08.244382 30455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "bcf9b8d5315b4c46b04a2249b3646798"
format_stamp: "Formatted at 2026-08-04 14:53:08 on dist-test-slave-92m1"
I20260804 14:53:08.248910 30455 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260804 14:53:08.251978 30472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:08.253600 30455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260804 14:53:08.253793 30455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "bcf9b8d5315b4c46b04a2249b3646798"
format_stamp: "Formatted at 2026-08-04 14:53:08 on dist-test-slave-92m1"
I20260804 14:53:08.253948 30455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:08.284111 30455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:08.284943 30455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:08.285188 30455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:08.285873 30455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:53:08.286939 30455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:53:08.287015 30455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:08.287104 30455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:53:08.287158 30455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:08.297650 30455 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.131:33693
I20260804 14:53:08.297667 30633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.131:33693 every 8 connection(s)
I20260804 14:53:08.298707 30455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260804 14:53:08.302456 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 30455
I20260804 14:53:08.302605 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260804 14:53:08.305235 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.132:0
--local_ip_for_outbound_sockets=127.24.6.132
--webserver_interface=127.24.6.132
--webserver_port=0
--tserver_master_addrs=127.24.6.190:42521
--builtin_ntp_servers=127.24.6.148:33757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20260804 14:53:08.313771 30634 heartbeater.cc:347] Connected to a master server at 127.24.6.190:42521
I20260804 14:53:08.314159 30634 heartbeater.cc:464] Registering TS with master...
I20260804 14:53:08.315056 30634 heartbeater.cc:511] Master 127.24.6.190:42521 requested a full tablet report, sending...
I20260804 14:53:08.316722 29992 ts_manager.cc:194] Registered new tserver with Master: bcf9b8d5315b4c46b04a2249b3646798 (127.24.6.131:33693)
I20260804 14:53:08.317459 29992 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.131:37165
W20260804 14:53:08.438107 30637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:08.438475 30637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:08.438516 30637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:08.443145 30637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:08.443324 30637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.132
I20260804 14:53:08.448495 30637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:33757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.24.6.132
--webserver_port=0
--tserver_master_addrs=127.24.6.190:42521
--never_fsync=true
--heap_profile_path=/tmp/kudu.30637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.132
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:08.449765 30637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:08.451344 30637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:08.462324 30647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:08.462467 30643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:08.462368 30644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:08.462612 30637 server_base.cc:1034] running on GCE node
I20260804 14:53:08.463346 30637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:08.464143 30637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:08.465358 30637 hybrid_clock.cc:648] HybridClock initialized: now 1785855188465319 us; error 67 us; skew 500 ppm
I20260804 14:53:08.467685 30637 webserver.cc:460] Webserver started at http://127.24.6.132:34787/ using document root <none> and password file <none>
I20260804 14:53:08.468340 30637 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:08.468442 30637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:08.468703 30637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:08.471024 30637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6"
format_stamp: "Formatted at 2026-08-04 14:53:08 on dist-test-slave-92m1"
I20260804 14:53:08.471762 30637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6"
format_stamp: "Formatted at 2026-08-04 14:53:08 on dist-test-slave-92m1"
I20260804 14:53:08.475966 30637 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260804 14:53:08.478758 30653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:08.480316 30637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260804 14:53:08.480438 30637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6"
format_stamp: "Formatted at 2026-08-04 14:53:08 on dist-test-slave-92m1"
I20260804 14:53:08.480592 30637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:08.494910 30637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:08.495824 30637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:08.496068 30637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:08.496760 30637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:53:08.497870 30637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:53:08.497947 30637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:08.498026 30637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:53:08.498072 30637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:08.508827 30637 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.132:45655
I20260804 14:53:08.508903 30822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.132:45655 every 8 connection(s)
I20260804 14:53:08.509922 30637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20260804 14:53:08.515690 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 30637
I20260804 14:53:08.515811 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20260804 14:53:08.518175 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.133:0
--local_ip_for_outbound_sockets=127.24.6.133
--webserver_interface=127.24.6.133
--webserver_port=0
--tserver_master_addrs=127.24.6.190:42521
--builtin_ntp_servers=127.24.6.148:33757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20260804 14:53:08.521870 30824 heartbeater.cc:347] Connected to a master server at 127.24.6.190:42521
I20260804 14:53:08.522181 30824 heartbeater.cc:464] Registering TS with master...
I20260804 14:53:08.522837 30824 heartbeater.cc:511] Master 127.24.6.190:42521 requested a full tablet report, sending...
I20260804 14:53:08.524194 29992 ts_manager.cc:194] Registered new tserver with Master: 7c45654f1b1d48cd8bf5e64bd66ddae6 (127.24.6.132:45655)
I20260804 14:53:08.524819 29992 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.132:60179
W20260804 14:53:08.649518 30827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:08.649883 30827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:08.649977 30827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:08.654337 30827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:08.654517 30827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.133
I20260804 14:53:08.659667 30827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:33757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.24.6.133
--webserver_port=0
--tserver_master_addrs=127.24.6.190:42521
--never_fsync=true
--heap_profile_path=/tmp/kudu.30827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.133
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:08.660857 30827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:08.662338 30827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:08.674109 30832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:08.674109 30834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:08.674165 30836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:08.674500 30827 server_base.cc:1034] running on GCE node
I20260804 14:53:08.674958 30827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:08.675738 30827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:08.676919 30827 hybrid_clock.cc:648] HybridClock initialized: now 1785855188676861 us; error 72 us; skew 500 ppm
I20260804 14:53:08.679287 30827 webserver.cc:460] Webserver started at http://127.24.6.133:41935/ using document root <none> and password file <none>
I20260804 14:53:08.679989 30827 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:08.680069 30827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:08.680258 30827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:08.682318 30827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "b395d77e158949d0a41c945ab5324571"
format_stamp: "Formatted at 2026-08-04 14:53:08 on dist-test-slave-92m1"
I20260804 14:53:08.682848 30827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "b395d77e158949d0a41c945ab5324571"
format_stamp: "Formatted at 2026-08-04 14:53:08 on dist-test-slave-92m1"
I20260804 14:53:08.687181 30827 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260804 14:53:08.690135 30843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:08.691598 30827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260804 14:53:08.691716 30827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b395d77e158949d0a41c945ab5324571"
format_stamp: "Formatted at 2026-08-04 14:53:08 on dist-test-slave-92m1"
I20260804 14:53:08.691820 30827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:08.704954 30827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:08.705698 30827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:08.705865 30827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:08.706472 30827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:53:08.707579 30827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:53:08.707630 30827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:08.707676 30827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:53:08.707692 30827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:08.719094 30827 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.133:40705
I20260804 14:53:08.719123 31012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.133:40705 every 8 connection(s)
I20260804 14:53:08.720192 30827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20260804 14:53:08.727972 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 30827
I20260804 14:53:08.728139 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20260804 14:53:08.733287 31013 heartbeater.cc:347] Connected to a master server at 127.24.6.190:42521
I20260804 14:53:08.733631 31013 heartbeater.cc:464] Registering TS with master...
I20260804 14:53:08.734251 31013 heartbeater.cc:511] Master 127.24.6.190:42521 requested a full tablet report, sending...
I20260804 14:53:08.735452 29992 ts_manager.cc:194] Registered new tserver with Master: b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705)
I20260804 14:53:08.736111 29992 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.133:52143
I20260804 14:53:08.743799 24602 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20260804 14:53:08.754952 29992 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60678:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20260804 14:53:08.758421 29992 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 6 tablet servers would be needed, 5 are available. Consider bringing up more tablet servers.
I20260804 14:53:08.781206 30175 tablet_service.cc:1467] Processing CreateTablet for tablet 6b4cb8c4a68f4801b05986e65f61a5ee (DEFAULT_TABLE table=TestTable [id=2552c6c899ae47b1aa3c7a81c12e7a20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:53:08.781484 30535 tablet_service.cc:1467] Processing CreateTablet for tablet 6b4cb8c4a68f4801b05986e65f61a5ee (DEFAULT_TABLE table=TestTable [id=2552c6c899ae47b1aa3c7a81c12e7a20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:53:08.781993 30723 tablet_service.cc:1467] Processing CreateTablet for tablet 6b4cb8c4a68f4801b05986e65f61a5ee (DEFAULT_TABLE table=TestTable [id=2552c6c899ae47b1aa3c7a81c12e7a20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:53:08.782670 30535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b4cb8c4a68f4801b05986e65f61a5ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:08.782544 30362 tablet_service.cc:1467] Processing CreateTablet for tablet 6b4cb8c4a68f4801b05986e65f61a5ee (DEFAULT_TABLE table=TestTable [id=2552c6c899ae47b1aa3c7a81c12e7a20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:53:08.783120 30723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b4cb8c4a68f4801b05986e65f61a5ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:08.783501 30362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b4cb8c4a68f4801b05986e65f61a5ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:08.784890 30915 tablet_service.cc:1467] Processing CreateTablet for tablet 6b4cb8c4a68f4801b05986e65f61a5ee (DEFAULT_TABLE table=TestTable [id=2552c6c899ae47b1aa3c7a81c12e7a20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:53:08.785975 30915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b4cb8c4a68f4801b05986e65f61a5ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:08.788168 30175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b4cb8c4a68f4801b05986e65f61a5ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:08.791380 31044 tablet_bootstrap.cc:492] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a: Bootstrap starting.
I20260804 14:53:08.791706 31045 tablet_bootstrap.cc:492] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798: Bootstrap starting.
I20260804 14:53:08.793931 31045 tablet_bootstrap.cc:654] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:08.793931 31044 tablet_bootstrap.cc:654] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:08.794845 31044 log.cc:824] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:08.794845 31045 log.cc:824] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:08.796965 31045 tablet_bootstrap.cc:492] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798: No bootstrap required, opened a new log
I20260804 14:53:08.796967 31044 tablet_bootstrap.cc:492] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a: No bootstrap required, opened a new log
I20260804 14:53:08.797187 31045 ts_tablet_manager.cc:1397] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260804 14:53:08.797187 31044 ts_tablet_manager.cc:1397] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260804 14:53:08.798928 31049 tablet_bootstrap.cc:492] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913: Bootstrap starting.
I20260804 14:53:08.801951 31045 raft_consensus.cc:348] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
I20260804 14:53:08.801918 31044 raft_consensus.cc:348] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
I20260804 14:53:08.802227 31045 raft_consensus.cc:374] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:53:08.802225 31044 raft_consensus.cc:374] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:53:08.802284 31045 raft_consensus.cc:729] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf9b8d5315b4c46b04a2249b3646798, State: Initialized, Role: FOLLOWER
I20260804 14:53:08.802284 31044 raft_consensus.cc:729] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3229081dc1fd49d4a4efd147a9ae069a, State: Initialized, Role: FOLLOWER
I20260804 14:53:08.802857 31044 consensus_queue.cc:260] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
I20260804 14:53:08.802857 31045 consensus_queue.cc:260] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
I20260804 14:53:08.803637 31049 tablet_bootstrap.cc:654] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:08.804013 31045 ts_tablet_manager.cc:1428] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798: Time spent starting tablet: real 0.007s user 0.000s sys 0.007s
I20260804 14:53:08.804004 31044 ts_tablet_manager.cc:1428] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260804 14:53:08.804401 30451 heartbeater.cc:503] Master 127.24.6.190:42521 was elected leader, sending a full tablet report...
I20260804 14:53:08.804761 31051 tablet_bootstrap.cc:492] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571: Bootstrap starting.
I20260804 14:53:08.804919 31049 log.cc:824] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:08.805043 31046 tablet_bootstrap.cc:492] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6: Bootstrap starting.
I20260804 14:53:08.805876 30634 heartbeater.cc:503] Master 127.24.6.190:42521 was elected leader, sending a full tablet report...
I20260804 14:53:08.807380 31051 tablet_bootstrap.cc:654] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:08.807385 31046 tablet_bootstrap.cc:654] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:08.808270 31046 log.cc:824] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:08.808334 31051 log.cc:824] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:08.810515 31046 tablet_bootstrap.cc:492] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6: No bootstrap required, opened a new log
I20260804 14:53:08.810758 31046 ts_tablet_manager.cc:1397] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260804 14:53:08.810820 31051 tablet_bootstrap.cc:492] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571: No bootstrap required, opened a new log
I20260804 14:53:08.811020 31051 ts_tablet_manager.cc:1397] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260804 14:53:08.811678 31049 tablet_bootstrap.cc:492] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913: No bootstrap required, opened a new log
I20260804 14:53:08.811877 31049 ts_tablet_manager.cc:1397] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.001s
I20260804 14:53:08.814742 31046 raft_consensus.cc:348] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
I20260804 14:53:08.815047 31046 raft_consensus.cc:374] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:53:08.815115 31046 raft_consensus.cc:729] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c45654f1b1d48cd8bf5e64bd66ddae6, State: Initialized, Role: FOLLOWER
I20260804 14:53:08.815629 31046 consensus_queue.cc:260] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
I20260804 14:53:08.816140 31051 raft_consensus.cc:348] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
I20260804 14:53:08.816367 30824 heartbeater.cc:503] Master 127.24.6.190:42521 was elected leader, sending a full tablet report...
I20260804 14:53:08.816439 31051 raft_consensus.cc:374] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:53:08.816532 31051 raft_consensus.cc:729] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b395d77e158949d0a41c945ab5324571, State: Initialized, Role: FOLLOWER
I20260804 14:53:08.816778 31046 ts_tablet_manager.cc:1428] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260804 14:53:08.816865 31049 raft_consensus.cc:348] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
I20260804 14:53:08.817046 31051 consensus_queue.cc:260] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
I20260804 14:53:08.817175 31049 raft_consensus.cc:374] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:53:08.817245 31049 raft_consensus.cc:729] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 172e2da08d1b44a78b70acbeb59a0913, State: Initialized, Role: FOLLOWER
I20260804 14:53:08.817704 31049 consensus_queue.cc:260] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
I20260804 14:53:08.817987 31013 heartbeater.cc:503] Master 127.24.6.190:42521 was elected leader, sending a full tablet report...
I20260804 14:53:08.818157 31051 ts_tablet_manager.cc:1428] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260804 14:53:08.819350 31049 ts_tablet_manager.cc:1428] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260804 14:53:08.819442 30272 heartbeater.cc:503] Master 127.24.6.190:42521 was elected leader, sending a full tablet report...
W20260804 14:53:08.827831 30273 tablet.cc:2369] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260804 14:53:08.847361 31062 raft_consensus.cc:482] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260804 14:53:08.847589 31062 raft_consensus.cc:504] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
I20260804 14:53:08.848927 31062 leader_election.cc:288] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 172e2da08d1b44a78b70acbeb59a0913 (127.24.6.129:38929), 7c45654f1b1d48cd8bf5e64bd66ddae6 (127.24.6.132:45655), bcf9b8d5315b4c46b04a2249b3646798 (127.24.6.131:33693), 3229081dc1fd49d4a4efd147a9ae069a (127.24.6.130:44109)
I20260804 14:53:08.855703 30390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "b395d77e158949d0a41c945ab5324571" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3229081dc1fd49d4a4efd147a9ae069a" is_pre_election: true
I20260804 14:53:08.855703 30750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "b395d77e158949d0a41c945ab5324571" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" is_pre_election: true
I20260804 14:53:08.855775 30570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "b395d77e158949d0a41c945ab5324571" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf9b8d5315b4c46b04a2249b3646798" is_pre_election: true
I20260804 14:53:08.855931 30201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "b395d77e158949d0a41c945ab5324571" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "172e2da08d1b44a78b70acbeb59a0913" is_pre_election: true
I20260804 14:53:08.856092 30390 raft_consensus.cc:2454] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b395d77e158949d0a41c945ab5324571 in term 0.
I20260804 14:53:08.856092 30750 raft_consensus.cc:2454] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b395d77e158949d0a41c945ab5324571 in term 0.
I20260804 14:53:08.856093 30570 raft_consensus.cc:2454] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b395d77e158949d0a41c945ab5324571 in term 0.
I20260804 14:53:08.856185 30201 raft_consensus.cc:2454] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b395d77e158949d0a41c945ab5324571 in term 0.
I20260804 14:53:08.856622 30848 leader_election.cc:302] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 172e2da08d1b44a78b70acbeb59a0913, 7c45654f1b1d48cd8bf5e64bd66ddae6, b395d77e158949d0a41c945ab5324571; no voters:
I20260804 14:53:08.856981 31062 raft_consensus.cc:2781] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260804 14:53:08.857081 31062 raft_consensus.cc:482] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260804 14:53:08.857129 31062 raft_consensus.cc:3037] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:08.858253 31062 raft_consensus.cc:504] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
I20260804 14:53:08.858827 31062 leader_election.cc:288] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571 [CANDIDATE]: Term 1 election: Requested vote from peers 172e2da08d1b44a78b70acbeb59a0913 (127.24.6.129:38929), 7c45654f1b1d48cd8bf5e64bd66ddae6 (127.24.6.132:45655), bcf9b8d5315b4c46b04a2249b3646798 (127.24.6.131:33693), 3229081dc1fd49d4a4efd147a9ae069a (127.24.6.130:44109)
I20260804 14:53:08.859175 30201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "b395d77e158949d0a41c945ab5324571" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "172e2da08d1b44a78b70acbeb59a0913"
I20260804 14:53:08.859360 30201 raft_consensus.cc:3037] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:08.859413 30750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "b395d77e158949d0a41c945ab5324571" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6"
I20260804 14:53:08.859413 30570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "b395d77e158949d0a41c945ab5324571" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf9b8d5315b4c46b04a2249b3646798"
I20260804 14:53:08.859545 30570 raft_consensus.cc:3037] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:08.859547 30750 raft_consensus.cc:3037] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:08.859719 30390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "b395d77e158949d0a41c945ab5324571" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3229081dc1fd49d4a4efd147a9ae069a"
I20260804 14:53:08.859849 30390 raft_consensus.cc:3037] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:08.860741 30201 raft_consensus.cc:2454] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b395d77e158949d0a41c945ab5324571 in term 1.
I20260804 14:53:08.860759 30750 raft_consensus.cc:2454] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b395d77e158949d0a41c945ab5324571 in term 1.
I20260804 14:53:08.860759 30570 raft_consensus.cc:2454] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b395d77e158949d0a41c945ab5324571 in term 1.
I20260804 14:53:08.861114 30848 leader_election.cc:302] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 7c45654f1b1d48cd8bf5e64bd66ddae6, b395d77e158949d0a41c945ab5324571, bcf9b8d5315b4c46b04a2249b3646798; no voters:
I20260804 14:53:08.861140 30390 raft_consensus.cc:2454] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b395d77e158949d0a41c945ab5324571 in term 1.
I20260804 14:53:08.861287 31062 raft_consensus.cc:2781] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:08.861589 31062 raft_consensus.cc:686] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571 [term 1 LEADER]: Becoming Leader. State: Replica: b395d77e158949d0a41c945ab5324571, State: Running, Role: LEADER
I20260804 14:53:08.861960 31062 consensus_queue.cc:237] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
I20260804 14:53:08.865737 29997 catalog_manager.cc:5495] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571 reported cstate change: term changed from 0 to 1, leader changed from <none> to b395d77e158949d0a41c945ab5324571 (127.24.6.133). New cstate: current_term: 1 leader_uuid: "b395d77e158949d0a41c945ab5324571" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } health_report { overall_health: UNKNOWN } } }
I20260804 14:53:08.874650 24602 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20260804 14:53:08.876351 24602 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 172e2da08d1b44a78b70acbeb59a0913 to finish bootstrapping
I20260804 14:53:08.883504 24602 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 3229081dc1fd49d4a4efd147a9ae069a to finish bootstrapping
I20260804 14:53:08.890635 24602 ts_itest-base.cc:233] Waiting for 1 tablets on tserver bcf9b8d5315b4c46b04a2249b3646798 to finish bootstrapping
I20260804 14:53:08.897523 24602 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 7c45654f1b1d48cd8bf5e64bd66ddae6 to finish bootstrapping
I20260804 14:53:08.904443 24602 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b395d77e158949d0a41c945ab5324571 to finish bootstrapping
I20260804 14:53:08.918529 24602 raft_consensus_election-itest.cc:397] Writing data to leader of 5-node config (5 alive)...
I20260804 14:53:08.936370 30570 raft_consensus.cc:1261] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 1 FOLLOWER]: Refusing update from remote peer b395d77e158949d0a41c945ab5324571: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260804 14:53:08.936370 30390 raft_consensus.cc:1261] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 1 FOLLOWER]: Refusing update from remote peer b395d77e158949d0a41c945ab5324571: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260804 14:53:08.936370 30750 raft_consensus.cc:1261] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 1 FOLLOWER]: Refusing update from remote peer b395d77e158949d0a41c945ab5324571: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260804 14:53:08.936370 30201 raft_consensus.cc:1261] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [term 1 FOLLOWER]: Refusing update from remote peer b395d77e158949d0a41c945ab5324571: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260804 14:53:08.937220 31062 consensus_queue.cc:1035] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260804 14:53:08.937422 31078 consensus_queue.cc:1035] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260804 14:53:08.937561 31079 consensus_queue.cc:1035] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571 [LEADER]: Connected to new peer: Peer: permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260804 14:53:08.937896 31070 consensus_queue.cc:1035] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260804 14:53:08.949223 31077 mvcc.cc:205] Tried to move back new op lower bound from 7314862853874135040 to 7314862853581922304. Current Snapshot: MvccSnapshot[applied={T|T < 7314862853874135040}]
I20260804 14:53:08.951105 31082 mvcc.cc:205] Tried to move back new op lower bound from 7314862853874135040 to 7314862853581922304. Current Snapshot: MvccSnapshot[applied={T|T < 7314862853874135040}]
I20260804 14:53:08.951656 31081 mvcc.cc:205] Tried to move back new op lower bound from 7314862853874135040 to 7314862853581922304. Current Snapshot: MvccSnapshot[applied={T|T < 7314862853874135040}]
I20260804 14:53:08.953652 31083 mvcc.cc:205] Tried to move back new op lower bound from 7314862853874135040 to 7314862853581922304. Current Snapshot: MvccSnapshot[applied={T|T < 7314862853874135040}]
I20260804 14:53:08.956925 31084 mvcc.cc:205] Tried to move back new op lower bound from 7314862853874135040 to 7314862853581922304. Current Snapshot: MvccSnapshot[applied={T|T < 7314862853874135040}]
I20260804 14:53:08.981181 24602 raft_consensus_election-itest.cc:408] Killing current leader b395d77e158949d0a41c945ab5324571...
I20260804 14:53:08.981307 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 30827
W20260804 14:53:08.999588 30302 connection.cc:535] server connection from 127.24.6.133:47269 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260804 14:53:09.000206 24602 raft_consensus_election-itest.cc:412] Waiting for new guy to be elected leader.
W20260804 14:53:09.000257 30476 connection.cc:535] server connection from 127.24.6.133:36759 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260804 14:53:09.000473 30656 connection.cc:535] server connection from 127.24.6.133:45847 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260804 14:53:09.002496 24602 test_util.cc:268] Using random seed: -802608904
I20260804 14:53:09.127400 24602 test_util.cc:268] Using random seed: -802484003
I20260804 14:53:09.335572 24602 test_util.cc:268] Using random seed: -802275831
I20260804 14:53:09.644035 24602 test_util.cc:268] Using random seed: -801967366
I20260804 14:53:10.052050 24602 test_util.cc:268] Using random seed: -801559350
I20260804 14:53:10.481945 31097 raft_consensus.cc:482] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b395d77e158949d0a41c945ab5324571)
I20260804 14:53:10.482149 31097 raft_consensus.cc:504] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
I20260804 14:53:10.482317 31099 raft_consensus.cc:482] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b395d77e158949d0a41c945ab5324571)
I20260804 14:53:10.482478 31099 raft_consensus.cc:504] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
I20260804 14:53:10.483870 31097 leader_election.cc:288] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7c45654f1b1d48cd8bf5e64bd66ddae6 (127.24.6.132:45655), bcf9b8d5315b4c46b04a2249b3646798 (127.24.6.131:33693), b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705), 3229081dc1fd49d4a4efd147a9ae069a (127.24.6.130:44109)
I20260804 14:53:10.486441 31099 leader_election.cc:288] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 172e2da08d1b44a78b70acbeb59a0913 (127.24.6.129:38929), bcf9b8d5315b4c46b04a2249b3646798 (127.24.6.131:33693), b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705), 3229081dc1fd49d4a4efd147a9ae069a (127.24.6.130:44109)
W20260804 14:53:10.486524 30110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.6.133:40705: connect: Connection refused (error 111)
W20260804 14:53:10.488871 30654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.6.133:40705: connect: Connection refused (error 111)
I20260804 14:53:10.490288 30750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "172e2da08d1b44a78b70acbeb59a0913" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" is_pre_election: true
I20260804 14:53:10.490473 30750 raft_consensus.cc:2454] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 172e2da08d1b44a78b70acbeb59a0913 in term 1.
W20260804 14:53:10.491014 30654 leader_election.cc:334] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705): Network error: Client connection negotiation failed: client connection to 127.24.6.133:40705: connect: Connection refused (error 111)
I20260804 14:53:10.491554 30201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "172e2da08d1b44a78b70acbeb59a0913" is_pre_election: true
I20260804 14:53:10.491744 30201 raft_consensus.cc:2454] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c45654f1b1d48cd8bf5e64bd66ddae6 in term 1.
I20260804 14:53:10.491997 31125 raft_consensus.cc:482] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b395d77e158949d0a41c945ab5324571)
I20260804 14:53:10.492146 31125 raft_consensus.cc:504] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
W20260804 14:53:10.492677 30110 leader_election.cc:334] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705): Network error: Client connection negotiation failed: client connection to 127.24.6.133:40705: connect: Connection refused (error 111)
I20260804 14:53:10.492731 30570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "bcf9b8d5315b4c46b04a2249b3646798" is_pre_election: true
I20260804 14:53:10.492870 30570 raft_consensus.cc:2454] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c45654f1b1d48cd8bf5e64bd66ddae6 in term 1.
I20260804 14:53:10.493238 30656 leader_election.cc:302] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 172e2da08d1b44a78b70acbeb59a0913, 7c45654f1b1d48cd8bf5e64bd66ddae6, bcf9b8d5315b4c46b04a2249b3646798; no voters: b395d77e158949d0a41c945ab5324571
I20260804 14:53:10.493834 31125 leader_election.cc:288] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 172e2da08d1b44a78b70acbeb59a0913 (127.24.6.129:38929), 7c45654f1b1d48cd8bf5e64bd66ddae6 (127.24.6.132:45655), bcf9b8d5315b4c46b04a2249b3646798 (127.24.6.131:33693), b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705)
I20260804 14:53:10.494140 31124 raft_consensus.cc:482] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b395d77e158949d0a41c945ab5324571)
I20260804 14:53:10.494252 31124 raft_consensus.cc:504] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
I20260804 14:53:10.493600 31099 raft_consensus.cc:2781] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260804 14:53:10.495011 31099 raft_consensus.cc:482] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b395d77e158949d0a41c945ab5324571)
I20260804 14:53:10.495131 31099 raft_consensus.cc:3037] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 1 FOLLOWER]: Advancing to term 2
I20260804 14:53:10.495733 31124 leader_election.cc:288] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 172e2da08d1b44a78b70acbeb59a0913 (127.24.6.129:38929), 7c45654f1b1d48cd8bf5e64bd66ddae6 (127.24.6.132:45655), b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705), 3229081dc1fd49d4a4efd147a9ae069a (127.24.6.130:44109)
I20260804 14:53:10.497516 30390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "172e2da08d1b44a78b70acbeb59a0913" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "3229081dc1fd49d4a4efd147a9ae069a" is_pre_election: true
I20260804 14:53:10.497675 30390 raft_consensus.cc:2454] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 172e2da08d1b44a78b70acbeb59a0913 in term 1.
I20260804 14:53:10.496644 31099 raft_consensus.cc:504] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
I20260804 14:53:10.497859 30570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "172e2da08d1b44a78b70acbeb59a0913" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "bcf9b8d5315b4c46b04a2249b3646798" is_pre_election: true
I20260804 14:53:10.497997 30570 raft_consensus.cc:2454] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 172e2da08d1b44a78b70acbeb59a0913 in term 1.
I20260804 14:53:10.498394 30117 leader_election.cc:302] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 172e2da08d1b44a78b70acbeb59a0913, 3229081dc1fd49d4a4efd147a9ae069a, 7c45654f1b1d48cd8bf5e64bd66ddae6; no voters: b395d77e158949d0a41c945ab5324571
I20260804 14:53:10.498647 30201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "172e2da08d1b44a78b70acbeb59a0913"
I20260804 14:53:10.498723 31097 raft_consensus.cc:2781] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260804 14:53:10.498788 30201 raft_consensus.cc:3037] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [term 1 FOLLOWER]: Advancing to term 2
I20260804 14:53:10.499749 30390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "3229081dc1fd49d4a4efd147a9ae069a" is_pre_election: true
I20260804 14:53:10.499895 30390 raft_consensus.cc:2454] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c45654f1b1d48cd8bf5e64bd66ddae6 in term 1.
I20260804 14:53:10.500159 30201 raft_consensus.cc:2454] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c45654f1b1d48cd8bf5e64bd66ddae6 in term 2.
I20260804 14:53:10.500276 31097 raft_consensus.cc:482] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260804 14:53:10.500344 31097 raft_consensus.cc:3037] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [term 2 FOLLOWER]: Advancing to term 3
W20260804 14:53:10.500800 30301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.6.133:40705: connect: Connection refused (error 111)
I20260804 14:53:10.501170 30570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "bcf9b8d5315b4c46b04a2249b3646798"
I20260804 14:53:10.501207 31099 leader_election.cc:288] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [CANDIDATE]: Term 2 election: Requested vote from peers 172e2da08d1b44a78b70acbeb59a0913 (127.24.6.129:38929), bcf9b8d5315b4c46b04a2249b3646798 (127.24.6.131:33693), b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705), 3229081dc1fd49d4a4efd147a9ae069a (127.24.6.130:44109)
I20260804 14:53:10.501291 30570 raft_consensus.cc:3037] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 1 FOLLOWER]: Advancing to term 2
I20260804 14:53:10.501636 30390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "3229081dc1fd49d4a4efd147a9ae069a"
I20260804 14:53:10.501612 31097 raft_consensus.cc:504] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
I20260804 14:53:10.501740 30390 raft_consensus.cc:3037] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 1 FOLLOWER]: Advancing to term 2
I20260804 14:53:10.502496 30570 raft_consensus.cc:2454] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c45654f1b1d48cd8bf5e64bd66ddae6 in term 2.
I20260804 14:53:10.502532 31097 leader_election.cc:288] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [CANDIDATE]: Term 3 election: Requested vote from peers 7c45654f1b1d48cd8bf5e64bd66ddae6 (127.24.6.132:45655), bcf9b8d5315b4c46b04a2249b3646798 (127.24.6.131:33693), b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705), 3229081dc1fd49d4a4efd147a9ae069a (127.24.6.130:44109)
I20260804 14:53:10.502854 30390 raft_consensus.cc:2454] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c45654f1b1d48cd8bf5e64bd66ddae6 in term 2.
I20260804 14:53:10.503232 30657 leader_election.cc:302] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 172e2da08d1b44a78b70acbeb59a0913, 3229081dc1fd49d4a4efd147a9ae069a, 7c45654f1b1d48cd8bf5e64bd66ddae6; no voters:
I20260804 14:53:10.503423 31099 raft_consensus.cc:2781] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 2 FOLLOWER]: Leader election won for term 2
I20260804 14:53:10.503545 30750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "172e2da08d1b44a78b70acbeb59a0913" candidate_term: 3 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6"
I20260804 14:53:10.503651 31099 raft_consensus.cc:686] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 2 LEADER]: Becoming Leader. State: Replica: 7c45654f1b1d48cd8bf5e64bd66ddae6, State: Running, Role: LEADER
I20260804 14:53:10.503720 30570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "172e2da08d1b44a78b70acbeb59a0913" candidate_term: 3 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "bcf9b8d5315b4c46b04a2249b3646798"
I20260804 14:53:10.503862 30570 raft_consensus.cc:3037] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 2 FOLLOWER]: Advancing to term 3
I20260804 14:53:10.504014 31099 consensus_queue.cc:237] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
I20260804 14:53:10.504431 30390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "172e2da08d1b44a78b70acbeb59a0913" candidate_term: 3 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "3229081dc1fd49d4a4efd147a9ae069a"
I20260804 14:53:10.504544 30390 raft_consensus.cc:3037] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 2 FOLLOWER]: Advancing to term 3
I20260804 14:53:10.504976 30570 raft_consensus.cc:2454] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 172e2da08d1b44a78b70acbeb59a0913 in term 3.
I20260804 14:53:10.505625 30390 raft_consensus.cc:2454] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 172e2da08d1b44a78b70acbeb59a0913 in term 3.
W20260804 14:53:10.506705 30110 leader_election.cc:334] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705): Network error: Client connection negotiation failed: client connection to 127.24.6.133:40705: connect: Connection refused (error 111)
I20260804 14:53:10.506929 30570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "3229081dc1fd49d4a4efd147a9ae069a" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "bcf9b8d5315b4c46b04a2249b3646798" is_pre_election: true
I20260804 14:53:10.507095 30570 raft_consensus.cc:2354] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3229081dc1fd49d4a4efd147a9ae069a for earlier term 2. Current term is 3.
I20260804 14:53:10.507269 29994 catalog_manager.cc:5495] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 reported cstate change: term changed from 1 to 2, leader changed from b395d77e158949d0a41c945ab5324571 (127.24.6.133) to 7c45654f1b1d48cd8bf5e64bd66ddae6 (127.24.6.132). New cstate: current_term: 2 leader_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } health_report { overall_health: UNKNOWN } } }
W20260804 14:53:10.507596 30654 leader_election.cc:334] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705): Network error: Client connection negotiation failed: client connection to 127.24.6.133:40705: connect: Connection refused (error 111)
I20260804 14:53:10.508421 30303 leader_election.cc:398] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [CANDIDATE]: Term 2 pre-election: Vote denied by peer bcf9b8d5315b4c46b04a2249b3646798 (127.24.6.131:33693) with higher term. Message: Invalid argument: T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3229081dc1fd49d4a4efd147a9ae069a for earlier term 2. Current term is 3.
I20260804 14:53:10.508499 30303 leader_election.cc:401] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260804 14:53:10.508741 31125 raft_consensus.cc:2726] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer bcf9b8d5315b4c46b04a2249b3646798 (127.24.6.131:33693) with higher term. Message: Invalid argument: T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3229081dc1fd49d4a4efd147a9ae069a for earlier term 2. Current term is 3.
W20260804 14:53:10.509737 30473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.6.133:40705: connect: Connection refused (error 111)
I20260804 14:53:10.511684 30117 leader_election.cc:302] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 5 voters: 3 yes votes; 2 no votes. yes voters: 172e2da08d1b44a78b70acbeb59a0913, 3229081dc1fd49d4a4efd147a9ae069a, bcf9b8d5315b4c46b04a2249b3646798; no voters: 7c45654f1b1d48cd8bf5e64bd66ddae6, b395d77e158949d0a41c945ab5324571
I20260804 14:53:10.511927 31097 raft_consensus.cc:2781] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [term 3 FOLLOWER]: Leader election won for term 3
I20260804 14:53:10.512199 31097 raft_consensus.cc:686] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [term 3 LEADER]: Becoming Leader. State: Replica: 172e2da08d1b44a78b70acbeb59a0913, State: Running, Role: LEADER
I20260804 14:53:10.512585 31097 consensus_queue.cc:237] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
I20260804 14:53:10.513875 30750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "3229081dc1fd49d4a4efd147a9ae069a" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" is_pre_election: true
W20260804 14:53:10.515187 30301 leader_election.cc:334] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705): Network error: Client connection negotiation failed: client connection to 127.24.6.133:40705: connect: Connection refused (error 111)
I20260804 14:53:10.516676 30390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "bcf9b8d5315b4c46b04a2249b3646798" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "3229081dc1fd49d4a4efd147a9ae069a" is_pre_election: true
I20260804 14:53:10.516791 30201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "3229081dc1fd49d4a4efd147a9ae069a" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "172e2da08d1b44a78b70acbeb59a0913" is_pre_election: true
I20260804 14:53:10.516852 30390 raft_consensus.cc:2354] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf9b8d5315b4c46b04a2249b3646798 for earlier term 2. Current term is 3.
I20260804 14:53:10.517369 30476 leader_election.cc:398] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 3229081dc1fd49d4a4efd147a9ae069a (127.24.6.130:44109) with higher term. Message: Invalid argument: T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf9b8d5315b4c46b04a2249b3646798 for earlier term 2. Current term is 3.
I20260804 14:53:10.517431 30476 leader_election.cc:401] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260804 14:53:10.517647 31124 raft_consensus.cc:2726] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 3229081dc1fd49d4a4efd147a9ae069a (127.24.6.130:44109) with higher term. Message: Invalid argument: T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf9b8d5315b4c46b04a2249b3646798 for earlier term 2. Current term is 3.
I20260804 14:53:10.518146 30201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "bcf9b8d5315b4c46b04a2249b3646798" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "172e2da08d1b44a78b70acbeb59a0913" is_pre_election: true
W20260804 14:53:10.519548 30473 leader_election.cc:334] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705): Network error: Client connection negotiation failed: client connection to 127.24.6.133:40705: connect: Connection refused (error 111)
I20260804 14:53:10.519661 29995 catalog_manager.cc:5495] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 reported cstate change: term changed from 2 to 3, leader changed from 7c45654f1b1d48cd8bf5e64bd66ddae6 (127.24.6.132) to 172e2da08d1b44a78b70acbeb59a0913 (127.24.6.129). New cstate: current_term: 3 leader_uuid: "172e2da08d1b44a78b70acbeb59a0913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } health_report { overall_health: UNKNOWN } } }
I20260804 14:53:10.520720 30750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "bcf9b8d5315b4c46b04a2249b3646798" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" is_pre_election: true
I20260804 14:53:10.559834 24602 raft_consensus_election-itest.cc:397] Writing data to leader of 5-node config (4 alive)...
W20260804 14:53:10.563112 31018 meta_cache.cc:302] tablet 6b4cb8c4a68f4801b05986e65f61a5ee: replica b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705) has failed: Network error: Client connection negotiation failed: client connection to 127.24.6.133:40705: connect: Connection refused (error 111)
I20260804 14:53:10.572739 30390 raft_consensus.cc:1261] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 3 FOLLOWER]: Refusing update from remote peer 172e2da08d1b44a78b70acbeb59a0913: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20260804 14:53:10.572734 30750 raft_consensus.cc:3032] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 2 LEADER]: Stepping down as leader of term 2
I20260804 14:53:10.572819 30570 raft_consensus.cc:1261] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 3 FOLLOWER]: Refusing update from remote peer 172e2da08d1b44a78b70acbeb59a0913: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20260804 14:53:10.572844 30750 raft_consensus.cc:729] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7c45654f1b1d48cd8bf5e64bd66ddae6, State: Running, Role: LEADER
I20260804 14:53:10.573063 30750 consensus_queue.cc:260] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.7, Last appended by leader: 7, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
W20260804 14:53:10.573366 30110 consensus_peers.cc:487] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 -> Peer b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705): Couldn't send request to peer b395d77e158949d0a41c945ab5324571. Status: Network error: Client connection negotiation failed: client connection to 127.24.6.133:40705: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260804 14:53:10.573465 30750 raft_consensus.cc:3037] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 2 FOLLOWER]: Advancing to term 3
I20260804 14:53:10.573438 31097 consensus_queue.cc:1035] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260804 14:53:10.573658 31160 consensus_queue.cc:1035] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260804 14:53:10.574970 30750 raft_consensus.cc:1261] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 3 FOLLOWER]: Refusing update from remote peer 172e2da08d1b44a78b70acbeb59a0913: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20260804 14:53:10.575753 31160 consensus_queue.cc:1035] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260804 14:53:10.575914 31160 consensus_queue.cc:1225] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [LEADER]: Peer 7c45654f1b1d48cd8bf5e64bd66ddae6 log is divergent from this leader: its last log entry 2.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260804 14:53:10.576718 30750 pending_rounds.cc:76] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6: Aborting all ops after (but not including) 6
I20260804 14:53:10.576783 30750 pending_rounds.cc:97] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6: Aborting uncommitted NO_OP operation due to leader change: 2.7
I20260804 14:53:10.576900 30750 raft_consensus.cc:2866] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260804 14:53:10.606674 24602 raft_consensus_election-itest.cc:408] Killing current leader 172e2da08d1b44a78b70acbeb59a0913...
I20260804 14:53:10.606803 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 30089
I20260804 14:53:10.623482 24602 raft_consensus_election-itest.cc:412] Waiting for new guy to be elected leader.
I20260804 14:53:10.626375 24602 test_util.cc:268] Using random seed: -800985027
I20260804 14:53:10.734331 24602 test_util.cc:268] Using random seed: -800877071
I20260804 14:53:10.942426 24602 test_util.cc:268] Using random seed: -800668975
I20260804 14:53:11.250029 24602 test_util.cc:268] Using random seed: -800361372
I20260804 14:53:11.657702 24602 test_util.cc:268] Using random seed: -799953701
I20260804 14:53:12.106719 31174 raft_consensus.cc:482] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 172e2da08d1b44a78b70acbeb59a0913)
I20260804 14:53:12.106771 31175 raft_consensus.cc:482] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 172e2da08d1b44a78b70acbeb59a0913)
I20260804 14:53:12.106910 31176 raft_consensus.cc:482] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 172e2da08d1b44a78b70acbeb59a0913)
I20260804 14:53:12.106889 31174 raft_consensus.cc:504] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
I20260804 14:53:12.106904 31175 raft_consensus.cc:504] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
I20260804 14:53:12.107048 31176 raft_consensus.cc:504] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
I20260804 14:53:12.107945 31175 leader_election.cc:288] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 172e2da08d1b44a78b70acbeb59a0913 (127.24.6.129:38929), 7c45654f1b1d48cd8bf5e64bd66ddae6 (127.24.6.132:45655), b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705), 3229081dc1fd49d4a4efd147a9ae069a (127.24.6.130:44109)
I20260804 14:53:12.108374 30390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "bcf9b8d5315b4c46b04a2249b3646798" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "3229081dc1fd49d4a4efd147a9ae069a" is_pre_election: true
I20260804 14:53:12.108395 30570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "bcf9b8d5315b4c46b04a2249b3646798" is_pre_election: true
I20260804 14:53:12.108536 30390 raft_consensus.cc:2454] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf9b8d5315b4c46b04a2249b3646798 in term 3.
I20260804 14:53:12.108534 30570 raft_consensus.cc:2454] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c45654f1b1d48cd8bf5e64bd66ddae6 in term 3.
I20260804 14:53:12.108831 30750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "bcf9b8d5315b4c46b04a2249b3646798" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" is_pre_election: true
I20260804 14:53:12.108865 31176 leader_election.cc:288] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 172e2da08d1b44a78b70acbeb59a0913 (127.24.6.129:38929), bcf9b8d5315b4c46b04a2249b3646798 (127.24.6.131:33693), b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705), 3229081dc1fd49d4a4efd147a9ae069a (127.24.6.130:44109)
I20260804 14:53:12.108964 30750 raft_consensus.cc:2454] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf9b8d5315b4c46b04a2249b3646798 in term 3.
I20260804 14:53:12.109268 30570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "3229081dc1fd49d4a4efd147a9ae069a" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "bcf9b8d5315b4c46b04a2249b3646798" is_pre_election: true
I20260804 14:53:12.109381 30570 raft_consensus.cc:2454] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3229081dc1fd49d4a4efd147a9ae069a in term 3.
I20260804 14:53:12.109550 31174 leader_election.cc:288] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 172e2da08d1b44a78b70acbeb59a0913 (127.24.6.129:38929), 7c45654f1b1d48cd8bf5e64bd66ddae6 (127.24.6.132:45655), bcf9b8d5315b4c46b04a2249b3646798 (127.24.6.131:33693), b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705)
I20260804 14:53:12.109457 30476 leader_election.cc:302] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3229081dc1fd49d4a4efd147a9ae069a, 7c45654f1b1d48cd8bf5e64bd66ddae6, bcf9b8d5315b4c46b04a2249b3646798; no voters:
I20260804 14:53:12.109854 30750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "3229081dc1fd49d4a4efd147a9ae069a" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" is_pre_election: true
I20260804 14:53:12.109972 30750 raft_consensus.cc:2454] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3229081dc1fd49d4a4efd147a9ae069a in term 3.
I20260804 14:53:12.110271 30390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "3229081dc1fd49d4a4efd147a9ae069a" is_pre_election: true
I20260804 14:53:12.110356 30304 leader_election.cc:302] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3229081dc1fd49d4a4efd147a9ae069a, 7c45654f1b1d48cd8bf5e64bd66ddae6, bcf9b8d5315b4c46b04a2249b3646798; no voters:
I20260804 14:53:12.110404 30390 raft_consensus.cc:2454] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c45654f1b1d48cd8bf5e64bd66ddae6 in term 3.
I20260804 14:53:12.110574 31174 raft_consensus.cc:2781] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260804 14:53:12.110631 31174 raft_consensus.cc:482] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 172e2da08d1b44a78b70acbeb59a0913)
I20260804 14:53:12.110674 31174 raft_consensus.cc:3037] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 3 FOLLOWER]: Advancing to term 4
I20260804 14:53:12.110704 30657 leader_election.cc:302] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3229081dc1fd49d4a4efd147a9ae069a, 7c45654f1b1d48cd8bf5e64bd66ddae6, bcf9b8d5315b4c46b04a2249b3646798; no voters:
W20260804 14:53:12.110770 30473 leader_election.cc:334] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 172e2da08d1b44a78b70acbeb59a0913 (127.24.6.129:38929): Network error: Client connection negotiation failed: client connection to 127.24.6.129:38929: connect: Connection refused (error 111)
I20260804 14:53:12.110874 31176 raft_consensus.cc:2781] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260804 14:53:12.110919 31176 raft_consensus.cc:482] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 172e2da08d1b44a78b70acbeb59a0913)
I20260804 14:53:12.110956 31176 raft_consensus.cc:3037] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 3 FOLLOWER]: Advancing to term 4
W20260804 14:53:12.111052 30473 leader_election.cc:334] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705): Network error: Client connection negotiation failed: client connection to 127.24.6.133:40705: connect: Connection refused (error 111)
W20260804 14:53:12.111254 30654 leader_election.cc:334] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705): Network error: Client connection negotiation failed: client connection to 127.24.6.133:40705: connect: Connection refused (error 111)
W20260804 14:53:12.111364 30654 leader_election.cc:334] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 172e2da08d1b44a78b70acbeb59a0913 (127.24.6.129:38929): Network error: Client connection negotiation failed: client connection to 127.24.6.129:38929: connect: Connection refused (error 111)
I20260804 14:53:12.111693 31175 raft_consensus.cc:2781] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260804 14:53:12.111759 31175 raft_consensus.cc:482] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 172e2da08d1b44a78b70acbeb59a0913)
I20260804 14:53:12.111817 31175 raft_consensus.cc:3037] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 3 FOLLOWER]: Advancing to term 4
I20260804 14:53:12.112020 31174 raft_consensus.cc:504] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
I20260804 14:53:12.112330 31176 raft_consensus.cc:504] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
I20260804 14:53:12.112584 31174 leader_election.cc:288] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [CANDIDATE]: Term 4 election: Requested vote from peers 172e2da08d1b44a78b70acbeb59a0913 (127.24.6.129:38929), 7c45654f1b1d48cd8bf5e64bd66ddae6 (127.24.6.132:45655), bcf9b8d5315b4c46b04a2249b3646798 (127.24.6.131:33693), b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705)
I20260804 14:53:12.112855 31176 leader_election.cc:288] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [CANDIDATE]: Term 4 election: Requested vote from peers 172e2da08d1b44a78b70acbeb59a0913 (127.24.6.129:38929), bcf9b8d5315b4c46b04a2249b3646798 (127.24.6.131:33693), b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705), 3229081dc1fd49d4a4efd147a9ae069a (127.24.6.130:44109)
I20260804 14:53:12.113242 30750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "3229081dc1fd49d4a4efd147a9ae069a" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6"
I20260804 14:53:12.113286 30570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "bcf9b8d5315b4c46b04a2249b3646798"
I20260804 14:53:12.113431 30750 raft_consensus.cc:2379] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3229081dc1fd49d4a4efd147a9ae069a in current term 4: Already voted for candidate 7c45654f1b1d48cd8bf5e64bd66ddae6 in this term.
I20260804 14:53:12.113643 30560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "3229081dc1fd49d4a4efd147a9ae069a" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "bcf9b8d5315b4c46b04a2249b3646798"
I20260804 14:53:12.113642 30390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "3229081dc1fd49d4a4efd147a9ae069a"
I20260804 14:53:12.113786 30390 raft_consensus.cc:2379] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7c45654f1b1d48cd8bf5e64bd66ddae6 in current term 4: Already voted for candidate 3229081dc1fd49d4a4efd147a9ae069a in this term.
I20260804 14:53:12.114010 31175 raft_consensus.cc:504] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
I20260804 14:53:12.114284 30570 raft_consensus.cc:2379] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7c45654f1b1d48cd8bf5e64bd66ddae6 in current term 4: Already voted for candidate bcf9b8d5315b4c46b04a2249b3646798 in this term.
W20260804 14:53:12.114408 30301 leader_election.cc:334] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 172e2da08d1b44a78b70acbeb59a0913 (127.24.6.129:38929): Network error: Client connection negotiation failed: client connection to 127.24.6.129:38929: connect: Connection refused (error 111)
W20260804 14:53:12.114574 30301 leader_election.cc:334] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705): Network error: Client connection negotiation failed: client connection to 127.24.6.133:40705: connect: Connection refused (error 111)
W20260804 14:53:12.114645 30654 leader_election.cc:334] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705): Network error: Client connection negotiation failed: client connection to 127.24.6.133:40705: connect: Connection refused (error 111)
I20260804 14:53:12.114704 30654 leader_election.cc:302] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 7c45654f1b1d48cd8bf5e64bd66ddae6; no voters: 3229081dc1fd49d4a4efd147a9ae069a, b395d77e158949d0a41c945ab5324571, bcf9b8d5315b4c46b04a2249b3646798
W20260804 14:53:12.114845 30654 leader_election.cc:334] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 172e2da08d1b44a78b70acbeb59a0913 (127.24.6.129:38929): Network error: Client connection negotiation failed: client connection to 127.24.6.129:38929: connect: Connection refused (error 111)
I20260804 14:53:12.114913 31175 leader_election.cc:288] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [CANDIDATE]: Term 4 election: Requested vote from peers 172e2da08d1b44a78b70acbeb59a0913 (127.24.6.129:38929), 7c45654f1b1d48cd8bf5e64bd66ddae6 (127.24.6.132:45655), b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705), 3229081dc1fd49d4a4efd147a9ae069a (127.24.6.130:44109)
I20260804 14:53:12.115253 30750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "bcf9b8d5315b4c46b04a2249b3646798" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6"
I20260804 14:53:12.115391 30750 raft_consensus.cc:2379] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf9b8d5315b4c46b04a2249b3646798 in current term 4: Already voted for candidate 7c45654f1b1d48cd8bf5e64bd66ddae6 in this term.
I20260804 14:53:12.115455 31176 raft_consensus.cc:2726] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260804 14:53:12.115715 30390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "bcf9b8d5315b4c46b04a2249b3646798" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "3229081dc1fd49d4a4efd147a9ae069a"
I20260804 14:53:12.115886 30390 raft_consensus.cc:2379] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf9b8d5315b4c46b04a2249b3646798 in current term 4: Already voted for candidate 3229081dc1fd49d4a4efd147a9ae069a in this term.
W20260804 14:53:12.116012 30473 leader_election.cc:334] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 172e2da08d1b44a78b70acbeb59a0913 (127.24.6.129:38929): Network error: Client connection negotiation failed: client connection to 127.24.6.129:38929: connect: Connection refused (error 111)
W20260804 14:53:12.116123 30473 leader_election.cc:334] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705): Network error: Client connection negotiation failed: client connection to 127.24.6.133:40705: connect: Connection refused (error 111)
I20260804 14:53:12.116174 30473 leader_election.cc:302] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: bcf9b8d5315b4c46b04a2249b3646798; no voters: 172e2da08d1b44a78b70acbeb59a0913, 7c45654f1b1d48cd8bf5e64bd66ddae6, b395d77e158949d0a41c945ab5324571
I20260804 14:53:12.116314 31175 raft_consensus.cc:2726] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260804 14:53:12.116492 30301 leader_election.cc:334] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 172e2da08d1b44a78b70acbeb59a0913 (127.24.6.129:38929): Network error: Client connection negotiation failed: client connection to 127.24.6.129:38929: connect: Connection refused (error 111)
I20260804 14:53:12.116560 30301 leader_election.cc:302] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 3229081dc1fd49d4a4efd147a9ae069a; no voters: 172e2da08d1b44a78b70acbeb59a0913, 7c45654f1b1d48cd8bf5e64bd66ddae6, bcf9b8d5315b4c46b04a2249b3646798
W20260804 14:53:12.116669 30301 leader_election.cc:334] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705): Network error: Client connection negotiation failed: client connection to 127.24.6.133:40705: connect: Connection refused (error 111)
I20260804 14:53:12.116688 31174 raft_consensus.cc:2726] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260804 14:53:12.165287 24602 test_util.cc:268] Using random seed: -799446114
I20260804 14:53:12.773841 24602 test_util.cc:268] Using random seed: -798837561
I20260804 14:53:13.481830 24602 test_util.cc:268] Using random seed: -798129572
I20260804 14:53:14.197851 31192 raft_consensus.cc:482] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260804 14:53:14.198014 31192 raft_consensus.cc:504] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
I20260804 14:53:14.198640 31192 leader_election.cc:288] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 172e2da08d1b44a78b70acbeb59a0913 (127.24.6.129:38929), 7c45654f1b1d48cd8bf5e64bd66ddae6 (127.24.6.132:45655), b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705), 3229081dc1fd49d4a4efd147a9ae069a (127.24.6.130:44109)
I20260804 14:53:14.199266 30390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "bcf9b8d5315b4c46b04a2249b3646798" candidate_term: 5 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "3229081dc1fd49d4a4efd147a9ae069a" is_pre_election: true
I20260804 14:53:14.199255 30750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "bcf9b8d5315b4c46b04a2249b3646798" candidate_term: 5 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" is_pre_election: true
I20260804 14:53:14.199433 30390 raft_consensus.cc:2454] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf9b8d5315b4c46b04a2249b3646798 in term 4.
I20260804 14:53:14.199455 30750 raft_consensus.cc:2454] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf9b8d5315b4c46b04a2249b3646798 in term 4.
I20260804 14:53:14.199900 30476 leader_election.cc:302] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3229081dc1fd49d4a4efd147a9ae069a, 7c45654f1b1d48cd8bf5e64bd66ddae6, bcf9b8d5315b4c46b04a2249b3646798; no voters:
I20260804 14:53:14.200173 31192 raft_consensus.cc:2781] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260804 14:53:14.200193 30473 leader_election.cc:334] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 172e2da08d1b44a78b70acbeb59a0913 (127.24.6.129:38929): Network error: Client connection negotiation failed: client connection to 127.24.6.129:38929: connect: Connection refused (error 111)
I20260804 14:53:14.200241 31192 raft_consensus.cc:482] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260804 14:53:14.200313 31192 raft_consensus.cc:3037] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 4 FOLLOWER]: Advancing to term 5
W20260804 14:53:14.200798 30473 leader_election.cc:334] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705): Network error: Client connection negotiation failed: client connection to 127.24.6.133:40705: connect: Connection refused (error 111)
I20260804 14:53:14.201340 31192 raft_consensus.cc:504] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
I20260804 14:53:14.201769 31192 leader_election.cc:288] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [CANDIDATE]: Term 5 election: Requested vote from peers 172e2da08d1b44a78b70acbeb59a0913 (127.24.6.129:38929), 7c45654f1b1d48cd8bf5e64bd66ddae6 (127.24.6.132:45655), b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705), 3229081dc1fd49d4a4efd147a9ae069a (127.24.6.130:44109)
I20260804 14:53:14.202248 30750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "bcf9b8d5315b4c46b04a2249b3646798" candidate_term: 5 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6"
I20260804 14:53:14.202288 30390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4cb8c4a68f4801b05986e65f61a5ee" candidate_uuid: "bcf9b8d5315b4c46b04a2249b3646798" candidate_term: 5 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "3229081dc1fd49d4a4efd147a9ae069a"
I20260804 14:53:14.202394 30750 raft_consensus.cc:3037] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 4 FOLLOWER]: Advancing to term 5
I20260804 14:53:14.202417 30390 raft_consensus.cc:3037] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 4 FOLLOWER]: Advancing to term 5
W20260804 14:53:14.202783 30473 leader_election.cc:334] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 172e2da08d1b44a78b70acbeb59a0913 (127.24.6.129:38929): Network error: Client connection negotiation failed: client connection to 127.24.6.129:38929: connect: Connection refused (error 111)
W20260804 14:53:14.203194 30473 leader_election.cc:334] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705): Network error: Client connection negotiation failed: client connection to 127.24.6.133:40705: connect: Connection refused (error 111)
I20260804 14:53:14.203544 30750 raft_consensus.cc:2454] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf9b8d5315b4c46b04a2249b3646798 in term 5.
I20260804 14:53:14.203544 30390 raft_consensus.cc:2454] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf9b8d5315b4c46b04a2249b3646798 in term 5.
I20260804 14:53:14.203894 30476 leader_election.cc:302] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 5 voters: 3 yes votes; 2 no votes. yes voters: 3229081dc1fd49d4a4efd147a9ae069a, 7c45654f1b1d48cd8bf5e64bd66ddae6, bcf9b8d5315b4c46b04a2249b3646798; no voters: 172e2da08d1b44a78b70acbeb59a0913, b395d77e158949d0a41c945ab5324571
I20260804 14:53:14.204100 31192 raft_consensus.cc:2781] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 5 FOLLOWER]: Leader election won for term 5
I20260804 14:53:14.204421 31192 raft_consensus.cc:686] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [term 5 LEADER]: Becoming Leader. State: Replica: bcf9b8d5315b4c46b04a2249b3646798, State: Running, Role: LEADER
I20260804 14:53:14.204795 31192 consensus_queue.cc:237] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
I20260804 14:53:14.207612 29995 catalog_manager.cc:5495] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 reported cstate change: term changed from 3 to 5, leader changed from 172e2da08d1b44a78b70acbeb59a0913 (127.24.6.129) to bcf9b8d5315b4c46b04a2249b3646798 (127.24.6.131). New cstate: current_term: 5 leader_uuid: "bcf9b8d5315b4c46b04a2249b3646798" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } health_report { overall_health: UNKNOWN } } }
I20260804 14:53:14.289683 24602 raft_consensus_election-itest.cc:397] Writing data to leader of 5-node config (3 alive)...
W20260804 14:53:14.293048 31018 meta_cache.cc:302] tablet 6b4cb8c4a68f4801b05986e65f61a5ee: replica 172e2da08d1b44a78b70acbeb59a0913 (127.24.6.129:38929) has failed: Network error: Client connection negotiation failed: client connection to 127.24.6.129:38929: connect: Connection refused (error 111)
W20260804 14:53:14.305609 30689 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36298: Illegal state: replica 7c45654f1b1d48cd8bf5e64bd66ddae6 is not leader of this config: current role FOLLOWER
I20260804 14:53:14.318764 30390 raft_consensus.cc:1261] T 6b4cb8c4a68f4801b05986e65f61a5ee P 3229081dc1fd49d4a4efd147a9ae069a [term 5 FOLLOWER]: Refusing update from remote peer bcf9b8d5315b4c46b04a2249b3646798: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 5 index: 14. (index mismatch)
I20260804 14:53:14.318841 30750 raft_consensus.cc:1261] T 6b4cb8c4a68f4801b05986e65f61a5ee P 7c45654f1b1d48cd8bf5e64bd66ddae6 [term 5 FOLLOWER]: Refusing update from remote peer bcf9b8d5315b4c46b04a2249b3646798: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 5 index: 14. (index mismatch)
I20260804 14:53:14.319420 31196 consensus_queue.cc:1035] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260804 14:53:14.319916 31216 consensus_queue.cc:1035] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260804 14:53:14.320113 30473 consensus_peers.cc:487] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 -> Peer b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705): Couldn't send request to peer b395d77e158949d0a41c945ab5324571. Status: Network error: Client connection negotiation failed: client connection to 127.24.6.133:40705: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260804 14:53:14.320277 30473 consensus_peers.cc:487] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 -> Peer 172e2da08d1b44a78b70acbeb59a0913 (127.24.6.129:38929): Couldn't send request to peer 172e2da08d1b44a78b70acbeb59a0913. Status: Network error: Client connection negotiation failed: client connection to 127.24.6.129:38929: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260804 14:53:14.350277 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.129:38929
--local_ip_for_outbound_sockets=127.24.6.129
--tserver_master_addrs=127.24.6.190:42521
--webserver_port=37977
--webserver_interface=127.24.6.129
--builtin_ntp_servers=127.24.6.148:33757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
W20260804 14:53:14.473099 31230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:14.473445 31230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:14.473516 31230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:14.477895 31230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:14.478068 31230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.129
I20260804 14:53:14.483430 31230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:33757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.129:38929
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.24.6.129
--webserver_port=37977
--tserver_master_addrs=127.24.6.190:42521
--never_fsync=true
--heap_profile_path=/tmp/kudu.31230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.129
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:14.484745 31230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:14.486310 31230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:14.496624 31237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:14.496624 31238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:14.496663 31241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:14.497095 31230 server_base.cc:1034] running on GCE node
I20260804 14:53:14.497625 31230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:14.498467 31230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:14.499667 31230 hybrid_clock.cc:648] HybridClock initialized: now 1785855194499656 us; error 33 us; skew 500 ppm
I20260804 14:53:14.505651 31230 webserver.cc:460] Webserver started at http://127.24.6.129:37977/ using document root <none> and password file <none>
I20260804 14:53:14.506475 31230 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:14.506628 31230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:14.511690 31230 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260804 14:53:14.514464 31247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:14.516016 31230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260804 14:53:14.516166 31230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "172e2da08d1b44a78b70acbeb59a0913"
format_stamp: "Formatted at 2026-08-04 14:53:07 on dist-test-slave-92m1"
I20260804 14:53:14.516819 31230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:14.550827 31230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:14.551694 31230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:14.551918 31230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:14.552630 31230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:53:14.554018 31255 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260804 14:53:14.558660 31230 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260804 14:53:14.558741 31230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260804 14:53:14.558826 31230 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260804 14:53:14.561050 31230 ts_tablet_manager.cc:610] Registered 1 tablets
I20260804 14:53:14.561129 31230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260804 14:53:14.561430 31255 tablet_bootstrap.cc:492] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913: Bootstrap starting.
I20260804 14:53:14.573375 31410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.129:38929 every 8 connection(s)
I20260804 14:53:14.573386 31230 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.129:38929
I20260804 14:53:14.575148 31230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260804 14:53:14.579788 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 31230
I20260804 14:53:14.580377 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.133:40705
--local_ip_for_outbound_sockets=127.24.6.133
--tserver_master_addrs=127.24.6.190:42521
--webserver_port=41935
--webserver_interface=127.24.6.133
--builtin_ntp_servers=127.24.6.148:33757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20260804 14:53:14.583371 31255 log.cc:824] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:14.591459 31411 heartbeater.cc:347] Connected to a master server at 127.24.6.190:42521
I20260804 14:53:14.591739 31411 heartbeater.cc:464] Registering TS with master...
I20260804 14:53:14.592425 31411 heartbeater.cc:511] Master 127.24.6.190:42521 requested a full tablet report, sending...
I20260804 14:53:14.595114 29997 ts_manager.cc:194] Re-registered known tserver with Master: 172e2da08d1b44a78b70acbeb59a0913 (127.24.6.129:38929)
I20260804 14:53:14.596227 29997 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.129:47579
I20260804 14:53:14.601279 31255 tablet_bootstrap.cc:492] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260804 14:53:14.601775 31255 tablet_bootstrap.cc:492] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913: Bootstrap complete.
I20260804 14:53:14.603034 31255 ts_tablet_manager.cc:1397] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913: Time spent bootstrapping tablet: real 0.042s user 0.030s sys 0.007s
I20260804 14:53:14.605909 31255 raft_consensus.cc:348] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
I20260804 14:53:14.606434 31255 raft_consensus.cc:729] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 172e2da08d1b44a78b70acbeb59a0913, State: Initialized, Role: FOLLOWER
I20260804 14:53:14.607017 31255 consensus_queue.cc:260] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
I20260804 14:53:14.608181 31411 heartbeater.cc:503] Master 127.24.6.190:42521 was elected leader, sending a full tablet report...
I20260804 14:53:14.608529 31255 ts_tablet_manager.cc:1428] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260804 14:53:14.714702 31419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:14.715045 31419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:14.715123 31419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:14.719594 31419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:14.719758 31419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.133
I20260804 14:53:14.725224 31419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:33757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.133:40705
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.24.6.133
--webserver_port=41935
--tserver_master_addrs=127.24.6.190:42521
--never_fsync=true
--heap_profile_path=/tmp/kudu.31419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.133
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:14.726475 31419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:14.728039 31419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:14.738816 31435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:14.738754 31437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:14.738763 31431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:14.739045 31419 server_base.cc:1034] running on GCE node
I20260804 14:53:14.739790 31419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:14.740586 31419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:14.741808 31419 hybrid_clock.cc:648] HybridClock initialized: now 1785855194741765 us; error 59 us; skew 500 ppm
I20260804 14:53:14.744204 31419 webserver.cc:460] Webserver started at http://127.24.6.133:41935/ using document root <none> and password file <none>
I20260804 14:53:14.744879 31419 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:14.745008 31419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:14.749806 31419 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260804 14:53:14.752413 31447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:14.753875 31419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260804 14:53:14.754070 31419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b395d77e158949d0a41c945ab5324571"
format_stamp: "Formatted at 2026-08-04 14:53:08 on dist-test-slave-92m1"
I20260804 14:53:14.754690 31419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:14.772389 31419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:14.773212 31419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:14.773439 31419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:14.774145 31419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:53:14.775619 31454 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260804 14:53:14.780396 31419 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260804 14:53:14.780514 31419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260804 14:53:14.780681 31419 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260804 14:53:14.782859 31419 ts_tablet_manager.cc:610] Registered 1 tablets
I20260804 14:53:14.782943 31419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260804 14:53:14.783303 31454 tablet_bootstrap.cc:492] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571: Bootstrap starting.
I20260804 14:53:14.795526 31419 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.133:40705
I20260804 14:53:14.795684 31590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.133:40705 every 8 connection(s)
I20260804 14:53:14.796767 31419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20260804 14:53:14.800836 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 31419
I20260804 14:53:14.801883 31454 log.cc:824] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:14.810009 31592 heartbeater.cc:347] Connected to a master server at 127.24.6.190:42521
I20260804 14:53:14.810508 31592 heartbeater.cc:464] Registering TS with master...
I20260804 14:53:14.811245 31592 heartbeater.cc:511] Master 127.24.6.190:42521 requested a full tablet report, sending...
I20260804 14:53:14.813278 29997 ts_manager.cc:194] Re-registered known tserver with Master: b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705)
I20260804 14:53:14.814093 29997 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.133:58751
I20260804 14:53:14.815536 31454 tablet_bootstrap.cc:492] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=50 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260804 14:53:14.816038 31454 tablet_bootstrap.cc:492] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571: Bootstrap complete.
I20260804 14:53:14.817476 31454 ts_tablet_manager.cc:1397] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571: Time spent bootstrapping tablet: real 0.035s user 0.024s sys 0.008s
I20260804 14:53:14.820816 31454 raft_consensus.cc:348] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
I20260804 14:53:14.821473 31454 raft_consensus.cc:729] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b395d77e158949d0a41c945ab5324571, State: Initialized, Role: FOLLOWER
I20260804 14:53:14.822058 31454 consensus_queue.cc:260] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172e2da08d1b44a78b70acbeb59a0913" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38929 } } peers { permanent_uuid: "7c45654f1b1d48cd8bf5e64bd66ddae6" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 45655 } } peers { permanent_uuid: "bcf9b8d5315b4c46b04a2249b3646798" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 33693 } } peers { permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 } } peers { permanent_uuid: "3229081dc1fd49d4a4efd147a9ae069a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 44109 } }
I20260804 14:53:14.822849 31592 heartbeater.cc:503] Master 127.24.6.190:42521 was elected leader, sending a full tablet report...
I20260804 14:53:14.823227 31454 ts_tablet_manager.cc:1428] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260804 14:53:14.868875 30362 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260804 14:53:14.871469 31507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260804 14:53:14.872432 31322 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260804 14:53:14.875481 30535 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260804 14:53:14.895816 31346 raft_consensus.cc:3037] T 6b4cb8c4a68f4801b05986e65f61a5ee P 172e2da08d1b44a78b70acbeb59a0913 [term 3 FOLLOWER]: Advancing to term 5
I20260804 14:53:14.904021 31648 mvcc.cc:205] Tried to move back new op lower bound from 7314862876044001280 to 7314862875465826304. Current Snapshot: MvccSnapshot[applied={T|T < 7314862875923173376}]
I20260804 14:53:14.907642 30723 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260804 14:53:14.912236 31531 raft_consensus.cc:3037] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571 [term 1 FOLLOWER]: Advancing to term 5
I20260804 14:53:14.915328 31531 raft_consensus.cc:1261] T 6b4cb8c4a68f4801b05986e65f61a5ee P b395d77e158949d0a41c945ab5324571 [term 5 FOLLOWER]: Refusing update from remote peer bcf9b8d5315b4c46b04a2249b3646798: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20260804 14:53:14.916134 31216 consensus_queue.cc:1037] T 6b4cb8c4a68f4801b05986e65f61a5ee P bcf9b8d5315b4c46b04a2249b3646798 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "b395d77e158949d0a41c945ab5324571" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 40705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 6, Time since last communication: 0.000s
I20260804 14:53:14.922823 31657 mvcc.cc:205] Tried to move back new op lower bound from 7314862860706643968 to 7314862860343808000. Current Snapshot: MvccSnapshot[applied={T|T < 7314862860579725312}]
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
75412ca9bcac463f8e54d370101ee5ff | 127.24.6.190:42521 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.24.6.148:33757 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
172e2da08d1b44a78b70acbeb59a0913 | 127.24.6.129:38929 | HEALTHY | <none> | 0 | 0
3229081dc1fd49d4a4efd147a9ae069a | 127.24.6.130:44109 | HEALTHY | <none> | 0 | 0
7c45654f1b1d48cd8bf5e64bd66ddae6 | 127.24.6.132:45655 | HEALTHY | <none> | 0 | 0
b395d77e158949d0a41c945ab5324571 | 127.24.6.133:40705 | HEALTHY | <none> | 0 | 0
bcf9b8d5315b4c46b04a2249b3646798 | 127.24.6.131:33693 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 5
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 5 server(s) checked
local_ip_for_outbound_sockets | 127.24.6.129 | experimental | 127.24.6.129:38929
local_ip_for_outbound_sockets | 127.24.6.130 | experimental | 127.24.6.130:44109
local_ip_for_outbound_sockets | 127.24.6.131 | experimental | 127.24.6.131:33693
local_ip_for_outbound_sockets | 127.24.6.132 | experimental | 127.24.6.132:45655
local_ip_for_outbound_sockets | 127.24.6.133 | experimental | 127.24.6.133:40705
never_fsync | true | unsafe,advanced | all 5 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 5 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 5 server(s) checked
server_dump_info_format | pb | hidden | all 5 server(s) checked
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.24.6.129:38929
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.24.6.130:44109
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.24.6.131:33693
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.24.6.132:45655
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.24.6.133:40705
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.24.6.148:33757 | all 5 server(s) checked
time_source | builtin | all 5 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 6 server(s) checked
Tablet Summary
Tablet 6b4cb8c4a68f4801b05986e65f61a5ee of table 'TestTable' is conflicted: 2 replicas' active configs disagree with the leader master's
172e2da08d1b44a78b70acbeb59a0913 (127.24.6.129:38929): RUNNING
7c45654f1b1d48cd8bf5e64bd66ddae6 (127.24.6.132:45655): RUNNING
bcf9b8d5315b4c46b04a2249b3646798 (127.24.6.131:33693): RUNNING [LEADER]
b395d77e158949d0a41c945ab5324571 (127.24.6.133:40705): RUNNING
3229081dc1fd49d4a4efd147a9ae069a (127.24.6.130:44109): RUNNING
All reported replicas are:
A = 172e2da08d1b44a78b70acbeb59a0913
B = 7c45654f1b1d48cd8bf5e64bd66ddae6
C = bcf9b8d5315b4c46b04a2249b3646798
D = b395d77e158949d0a41c945ab5324571
E = 3229081dc1fd49d4a4efd147a9ae069a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------------------+--------------+--------------+------------
master | A B C* D E | | | Yes
A | A B C D E | 3 | -1 | Yes
B | A B C* D E | 5 | -1 | Yes
C | A B C* D E | 5 | -1 | Yes
D | A B C D E | 1 | -1 | Yes
E | A B C* D E | 5 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+--------------------+---------------+---------+------------+------------------+-------------
TestTable | 5 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 5
Tables | 1
Tablets | 1
Replicas | 5
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260804 14:53:14.932251 24602 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260804 14:53:15.143555 30723 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260804 14:53:15.144804 31507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260804 14:53:15.144892 31322 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260804 14:53:15.146100 30362 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260804 14:53:15.148298 30535 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
75412ca9bcac463f8e54d370101ee5ff | 127.24.6.190:42521 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.24.6.148:33757 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
172e2da08d1b44a78b70acbeb59a0913 | 127.24.6.129:38929 | HEALTHY | <none> | 0 | 0
3229081dc1fd49d4a4efd147a9ae069a | 127.24.6.130:44109 | HEALTHY | <none> | 0 | 0
7c45654f1b1d48cd8bf5e64bd66ddae6 | 127.24.6.132:45655 | HEALTHY | <none> | 0 | 0
b395d77e158949d0a41c945ab5324571 | 127.24.6.133:40705 | HEALTHY | <none> | 0 | 0
bcf9b8d5315b4c46b04a2249b3646798 | 127.24.6.131:33693 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 5
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 5 server(s) checked
local_ip_for_outbound_sockets | 127.24.6.129 | experimental | 127.24.6.129:38929
local_ip_for_outbound_sockets | 127.24.6.130 | experimental | 127.24.6.130:44109
local_ip_for_outbound_sockets | 127.24.6.131 | experimental | 127.24.6.131:33693
local_ip_for_outbound_sockets | 127.24.6.132 | experimental | 127.24.6.132:45655
local_ip_for_outbound_sockets | 127.24.6.133 | experimental | 127.24.6.133:40705
never_fsync | true | unsafe,advanced | all 5 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 5 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 5 server(s) checked
server_dump_info_format | pb | hidden | all 5 server(s) checked
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.24.6.129:38929
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.24.6.130:44109
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.24.6.131:33693
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.24.6.132:45655
server_dump_info_path | /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.24.6.133:40705
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.24.6.148:33757 | all 5 server(s) checked
time_source | builtin | all 5 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 6 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 5 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 5
Tables | 1
Tablets | 1
Replicas | 5
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260804 14:53:15.185315 24602 log_verifier.cc:126] Checking tablet 6b4cb8c4a68f4801b05986e65f61a5ee
I20260804 14:53:15.235903 24602 log_verifier.cc:177] Verified matching terms for 18 ops in tablet 6b4cb8c4a68f4801b05986e65f61a5ee
I20260804 14:53:15.491297 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 31230
I20260804 14:53:15.501087 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 30276
I20260804 14:53:15.516139 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 30455
I20260804 14:53:15.531551 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 30637
I20260804 14:53:15.545194 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 31419
I20260804 14:53:15.556277 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 29960
2026-08-04T14:53:15Z chronyd exiting
[ OK ] RaftConsensusElectionITest.AutomaticLeaderElection (8254 ms)
[ RUN ] RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica
I20260804 14:53:15.594398 24602 test_util.cc:268] Using random seed: -796017001
I20260804 14:53:15.595384 24602 ts_itest-base.cc:112] Starting cluster with:
I20260804 14:53:15.595455 24602 ts_itest-base.cc:113] --------------
I20260804 14:53:15.595476 24602 ts_itest-base.cc:114] 1 tablet servers
I20260804 14:53:15.595504 24602 ts_itest-base.cc:115] 1 replicas per TS
I20260804 14:53:15.595530 24602 ts_itest-base.cc:116] --------------
2026-08-04T14:53:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-04T14:53:15Z Disabled control of system clock
I20260804 14:53:15.607831 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.6.190:35957
--webserver_interface=127.24.6.190
--webserver_port=0
--builtin_ntp_servers=127.24.6.148:35721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.6.190:35957 with env {}
W20260804 14:53:15.736037 31731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:15.736397 31731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:15.736477 31731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:15.741357 31731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260804 14:53:15.741524 31731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:15.741561 31731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260804 14:53:15.741604 31731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260804 14:53:15.747289 31731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:35721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.6.190:35957
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.6.190:35957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.24.6.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.31731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:15.748633 31731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:15.750237 31731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:15.760207 31739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:15.760214 31736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:15.760231 31737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:15.760473 31731 server_base.cc:1034] running on GCE node
I20260804 14:53:15.761299 31731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:15.762632 31731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:15.763844 31731 hybrid_clock.cc:648] HybridClock initialized: now 1785855195763834 us; error 37 us; skew 500 ppm
I20260804 14:53:15.766002 31731 webserver.cc:460] Webserver started at http://127.24.6.190:34721/ using document root <none> and password file <none>
I20260804 14:53:15.766697 31731 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:15.766805 31731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:15.767066 31731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:15.769354 31731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "1751dde17f4d4c2ab1a8c5db4860f87d"
format_stamp: "Formatted at 2026-08-04 14:53:15 on dist-test-slave-92m1"
I20260804 14:53:15.769985 31731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "1751dde17f4d4c2ab1a8c5db4860f87d"
format_stamp: "Formatted at 2026-08-04 14:53:15 on dist-test-slave-92m1"
I20260804 14:53:15.774261 31731 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260804 14:53:15.777190 31746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:15.778632 31731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260804 14:53:15.778913 31731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
uuid: "1751dde17f4d4c2ab1a8c5db4860f87d"
format_stamp: "Formatted at 2026-08-04 14:53:15 on dist-test-slave-92m1"
I20260804 14:53:15.779048 31731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:15.794831 31731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:15.795753 31731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:15.795992 31731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:15.806036 31731 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.190:35957
I20260804 14:53:15.806104 31824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.190:35957 every 8 connection(s)
I20260804 14:53:15.807228 31731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260804 14:53:15.811071 31825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:15.817262 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 31731
I20260804 14:53:15.817437 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260804 14:53:15.818320 31825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1751dde17f4d4c2ab1a8c5db4860f87d: Bootstrap starting.
I20260804 14:53:15.821700 31825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1751dde17f4d4c2ab1a8c5db4860f87d: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:15.822633 31825 log.cc:824] T 00000000000000000000000000000000 P 1751dde17f4d4c2ab1a8c5db4860f87d: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:15.825053 31825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1751dde17f4d4c2ab1a8c5db4860f87d: No bootstrap required, opened a new log
I20260804 14:53:15.829934 31825 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1751dde17f4d4c2ab1a8c5db4860f87d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1751dde17f4d4c2ab1a8c5db4860f87d" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 35957 } }
I20260804 14:53:15.830333 31825 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1751dde17f4d4c2ab1a8c5db4860f87d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:53:15.830436 31825 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1751dde17f4d4c2ab1a8c5db4860f87d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1751dde17f4d4c2ab1a8c5db4860f87d, State: Initialized, Role: FOLLOWER
I20260804 14:53:15.830976 31825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1751dde17f4d4c2ab1a8c5db4860f87d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1751dde17f4d4c2ab1a8c5db4860f87d" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 35957 } }
I20260804 14:53:15.831167 31825 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1751dde17f4d4c2ab1a8c5db4860f87d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260804 14:53:15.831279 31825 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1751dde17f4d4c2ab1a8c5db4860f87d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260804 14:53:15.831439 31825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1751dde17f4d4c2ab1a8c5db4860f87d [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:15.832544 31825 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1751dde17f4d4c2ab1a8c5db4860f87d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1751dde17f4d4c2ab1a8c5db4860f87d" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 35957 } }
I20260804 14:53:15.833006 31825 leader_election.cc:302] T 00000000000000000000000000000000 P 1751dde17f4d4c2ab1a8c5db4860f87d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1751dde17f4d4c2ab1a8c5db4860f87d; no voters:
I20260804 14:53:15.833470 31825 leader_election.cc:288] T 00000000000000000000000000000000 P 1751dde17f4d4c2ab1a8c5db4860f87d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260804 14:53:15.833559 31832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1751dde17f4d4c2ab1a8c5db4860f87d [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:15.833791 31832 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1751dde17f4d4c2ab1a8c5db4860f87d [term 1 LEADER]: Becoming Leader. State: Replica: 1751dde17f4d4c2ab1a8c5db4860f87d, State: Running, Role: LEADER
I20260804 14:53:15.834144 31832 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1751dde17f4d4c2ab1a8c5db4860f87d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1751dde17f4d4c2ab1a8c5db4860f87d" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 35957 } }
I20260804 14:53:15.835076 31825 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1751dde17f4d4c2ab1a8c5db4860f87d [sys.catalog]: configured and running, proceeding with master startup.
I20260804 14:53:15.836172 31833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1751dde17f4d4c2ab1a8c5db4860f87d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1751dde17f4d4c2ab1a8c5db4860f87d. Latest consensus state: current_term: 1 leader_uuid: "1751dde17f4d4c2ab1a8c5db4860f87d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1751dde17f4d4c2ab1a8c5db4860f87d" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 35957 } } }
I20260804 14:53:15.836338 31833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1751dde17f4d4c2ab1a8c5db4860f87d [sys.catalog]: This master's current role is: LEADER
I20260804 14:53:15.836818 31834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1751dde17f4d4c2ab1a8c5db4860f87d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1751dde17f4d4c2ab1a8c5db4860f87d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1751dde17f4d4c2ab1a8c5db4860f87d" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 35957 } } }
I20260804 14:53:15.836977 31834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1751dde17f4d4c2ab1a8c5db4860f87d [sys.catalog]: This master's current role is: LEADER
I20260804 14:53:15.837832 31842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260804 14:53:15.841363 31842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260804 14:53:15.848841 31842 catalog_manager.cc:1347] Generated new cluster ID: 4064ef353e33486db1ff5b2ee021b447
I20260804 14:53:15.848961 31842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260804 14:53:15.865104 31842 catalog_manager.cc:1370] Generated new certificate authority record
I20260804 14:53:15.866190 31842 catalog_manager.cc:1504] Loading token signing keys...
I20260804 14:53:15.871181 31842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1751dde17f4d4c2ab1a8c5db4860f87d: Generated new TSK 0
I20260804 14:53:15.872001 31842 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260804 14:53:15.882079 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.129:0
--local_ip_for_outbound_sockets=127.24.6.129
--webserver_interface=127.24.6.129
--webserver_port=0
--tserver_master_addrs=127.24.6.190:35957
--builtin_ntp_servers=127.24.6.148:35721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
W20260804 14:53:16.008600 31856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:16.008906 31856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:16.008944 31856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:16.013362 31856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:16.013487 31856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.129
I20260804 14:53:16.018489 31856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:35721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.24.6.129
--webserver_port=0
--tserver_master_addrs=127.24.6.190:35957
--never_fsync=true
--heap_profile_path=/tmp/kudu.31856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.129
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:16.019783 31856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:16.021363 31856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:16.032215 31865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:16.032393 31856 server_base.cc:1034] running on GCE node
W20260804 14:53:16.032217 31864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:16.032234 31867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:16.033032 31856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:16.033775 31856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:16.034952 31856 hybrid_clock.cc:648] HybridClock initialized: now 1785855196034915 us; error 44 us; skew 500 ppm
I20260804 14:53:16.037454 31856 webserver.cc:460] Webserver started at http://127.24.6.129:38209/ using document root <none> and password file <none>
I20260804 14:53:16.038123 31856 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:16.038198 31856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:16.038395 31856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:16.040892 31856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e176d336c7f941bcac1e63981afa6637"
format_stamp: "Formatted at 2026-08-04 14:53:16 on dist-test-slave-92m1"
I20260804 14:53:16.041648 31856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e176d336c7f941bcac1e63981afa6637"
format_stamp: "Formatted at 2026-08-04 14:53:16 on dist-test-slave-92m1"
I20260804 14:53:16.046535 31856 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260804 14:53:16.049537 31874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:16.051368 31856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260804 14:53:16.051544 31856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e176d336c7f941bcac1e63981afa6637"
format_stamp: "Formatted at 2026-08-04 14:53:16 on dist-test-slave-92m1"
I20260804 14:53:16.051690 31856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:16.061923 31856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:16.062723 31856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:16.062953 31856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:16.063659 31856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:53:16.064730 31856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:53:16.064806 31856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:16.064883 31856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:53:16.064930 31856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:16.074510 31856 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.129:43311
I20260804 14:53:16.074534 32034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.129:43311 every 8 connection(s)
I20260804 14:53:16.075572 31856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260804 14:53:16.081522 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 31856
I20260804 14:53:16.081633 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260804 14:53:16.087141 32035 heartbeater.cc:347] Connected to a master server at 127.24.6.190:35957
I20260804 14:53:16.087486 32035 heartbeater.cc:464] Registering TS with master...
I20260804 14:53:16.088083 32035 heartbeater.cc:511] Master 127.24.6.190:35957 requested a full tablet report, sending...
I20260804 14:53:16.089728 31767 ts_manager.cc:194] Registered new tserver with Master: e176d336c7f941bcac1e63981afa6637 (127.24.6.129:43311)
I20260804 14:53:16.090952 31767 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.129:39663
I20260804 14:53:16.096323 24602 external_mini_cluster.cc:930] 1 TS(s) registered with all masters
I20260804 14:53:16.106796 31767 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54806:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20260804 14:53:16.127693 31942 tablet_service.cc:1467] Processing CreateTablet for tablet c7037a96c4b7418a864e8625246d37fd (DEFAULT_TABLE table=TestTable [id=848c1350c8164885b75974bc054ee1c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:53:16.128811 31942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7037a96c4b7418a864e8625246d37fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:16.135571 32051 tablet_bootstrap.cc:492] T c7037a96c4b7418a864e8625246d37fd P e176d336c7f941bcac1e63981afa6637: Bootstrap starting.
I20260804 14:53:16.138494 32051 tablet_bootstrap.cc:654] T c7037a96c4b7418a864e8625246d37fd P e176d336c7f941bcac1e63981afa6637: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:16.139334 32051 log.cc:824] T c7037a96c4b7418a864e8625246d37fd P e176d336c7f941bcac1e63981afa6637: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:16.141374 32051 tablet_bootstrap.cc:492] T c7037a96c4b7418a864e8625246d37fd P e176d336c7f941bcac1e63981afa6637: No bootstrap required, opened a new log
I20260804 14:53:16.141626 32051 ts_tablet_manager.cc:1397] T c7037a96c4b7418a864e8625246d37fd P e176d336c7f941bcac1e63981afa6637: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260804 14:53:16.145701 32051 raft_consensus.cc:348] T c7037a96c4b7418a864e8625246d37fd P e176d336c7f941bcac1e63981afa6637 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e176d336c7f941bcac1e63981afa6637" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43311 } }
I20260804 14:53:16.146004 32051 raft_consensus.cc:374] T c7037a96c4b7418a864e8625246d37fd P e176d336c7f941bcac1e63981afa6637 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:53:16.146101 32051 raft_consensus.cc:729] T c7037a96c4b7418a864e8625246d37fd P e176d336c7f941bcac1e63981afa6637 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e176d336c7f941bcac1e63981afa6637, State: Initialized, Role: FOLLOWER
I20260804 14:53:16.146577 32051 consensus_queue.cc:260] T c7037a96c4b7418a864e8625246d37fd P e176d336c7f941bcac1e63981afa6637 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e176d336c7f941bcac1e63981afa6637" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43311 } }
I20260804 14:53:16.146741 32051 raft_consensus.cc:388] T c7037a96c4b7418a864e8625246d37fd P e176d336c7f941bcac1e63981afa6637 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260804 14:53:16.146836 32051 raft_consensus.cc:482] T c7037a96c4b7418a864e8625246d37fd P e176d336c7f941bcac1e63981afa6637 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260804 14:53:16.146967 32051 raft_consensus.cc:3037] T c7037a96c4b7418a864e8625246d37fd P e176d336c7f941bcac1e63981afa6637 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:16.148005 32051 raft_consensus.cc:504] T c7037a96c4b7418a864e8625246d37fd P e176d336c7f941bcac1e63981afa6637 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e176d336c7f941bcac1e63981afa6637" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43311 } }
I20260804 14:53:16.148393 32051 leader_election.cc:302] T c7037a96c4b7418a864e8625246d37fd P e176d336c7f941bcac1e63981afa6637 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e176d336c7f941bcac1e63981afa6637; no voters:
I20260804 14:53:16.148777 32051 leader_election.cc:288] T c7037a96c4b7418a864e8625246d37fd P e176d336c7f941bcac1e63981afa6637 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260804 14:53:16.148957 32053 raft_consensus.cc:2781] T c7037a96c4b7418a864e8625246d37fd P e176d336c7f941bcac1e63981afa6637 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:16.149201 32053 raft_consensus.cc:686] T c7037a96c4b7418a864e8625246d37fd P e176d336c7f941bcac1e63981afa6637 [term 1 LEADER]: Becoming Leader. State: Replica: e176d336c7f941bcac1e63981afa6637, State: Running, Role: LEADER
I20260804 14:53:16.149509 32035 heartbeater.cc:503] Master 127.24.6.190:35957 was elected leader, sending a full tablet report...
I20260804 14:53:16.149534 32053 consensus_queue.cc:237] T c7037a96c4b7418a864e8625246d37fd P e176d336c7f941bcac1e63981afa6637 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e176d336c7f941bcac1e63981afa6637" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43311 } }
I20260804 14:53:16.149935 32051 ts_tablet_manager.cc:1428] T c7037a96c4b7418a864e8625246d37fd P e176d336c7f941bcac1e63981afa6637: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260804 14:53:16.153661 31767 catalog_manager.cc:5495] T c7037a96c4b7418a864e8625246d37fd P e176d336c7f941bcac1e63981afa6637 reported cstate change: term changed from 0 to 1, leader changed from <none> to e176d336c7f941bcac1e63981afa6637 (127.24.6.129). New cstate: current_term: 1 leader_uuid: "e176d336c7f941bcac1e63981afa6637" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e176d336c7f941bcac1e63981afa6637" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 43311 } health_report { overall_health: HEALTHY } } }
I20260804 14:53:16.163118 24602 external_mini_cluster.cc:930] 1 TS(s) registered with all masters
I20260804 14:53:16.164757 24602 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e176d336c7f941bcac1e63981afa6637 to finish bootstrapping
I20260804 14:53:16.177826 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 31856
I20260804 14:53:16.193769 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 31731
2026-08-04T14:53:16Z chronyd exiting
[ OK ] RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica (625 ms)
[ RUN ] RaftConsensusElectionITest.LeaderStepDown
I20260804 14:53:16.220243 24602 test_util.cc:268] Using random seed: -795391155
I20260804 14:53:16.221249 24602 ts_itest-base.cc:112] Starting cluster with:
I20260804 14:53:16.221294 24602 ts_itest-base.cc:113] --------------
I20260804 14:53:16.221307 24602 ts_itest-base.cc:114] 3 tablet servers
I20260804 14:53:16.221320 24602 ts_itest-base.cc:115] 3 replicas per TS
I20260804 14:53:16.221369 24602 ts_itest-base.cc:116] --------------
2026-08-04T14:53:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-04T14:53:16Z Disabled control of system clock
I20260804 14:53:16.234277 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.6.190:40781
--webserver_interface=127.24.6.190
--webserver_port=0
--builtin_ntp_servers=127.24.6.148:37271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.6.190:40781
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20260804 14:53:16.359148 32070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:16.359573 32070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:16.359644 32070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:16.364148 32070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260804 14:53:16.364271 32070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:16.364306 32070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260804 14:53:16.364349 32070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260804 14:53:16.369547 32070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:37271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.24.6.190:40781
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.6.190:40781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.24.6.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.32070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:16.370889 32070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:16.372546 32070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:16.381953 32080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:16.381953 32077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:16.382081 32070 server_base.cc:1034] running on GCE node
W20260804 14:53:16.381979 32082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:16.383092 32070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:16.384495 32070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:16.385717 32070 hybrid_clock.cc:648] HybridClock initialized: now 1785855196385665 us; error 72 us; skew 500 ppm
I20260804 14:53:16.388118 32070 webserver.cc:460] Webserver started at http://127.24.6.190:42821/ using document root <none> and password file <none>
I20260804 14:53:16.388799 32070 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:16.388907 32070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:16.389175 32070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:16.391484 32070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "0e8182d4ea604f7b9ee1d68918f525d8"
format_stamp: "Formatted at 2026-08-04 14:53:16 on dist-test-slave-92m1"
I20260804 14:53:16.392145 32070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "0e8182d4ea604f7b9ee1d68918f525d8"
format_stamp: "Formatted at 2026-08-04 14:53:16 on dist-test-slave-92m1"
I20260804 14:53:16.396531 32070 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260804 14:53:16.399595 32088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:16.401178 32070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260804 14:53:16.401466 32070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
uuid: "0e8182d4ea604f7b9ee1d68918f525d8"
format_stamp: "Formatted at 2026-08-04 14:53:16 on dist-test-slave-92m1"
I20260804 14:53:16.401620 32070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:16.419013 32070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:16.419902 32070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:16.420156 32070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:16.428853 32070 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.190:40781
I20260804 14:53:16.428865 32171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.190:40781 every 8 connection(s)
I20260804 14:53:16.430001 32070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260804 14:53:16.432528 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 32070
I20260804 14:53:16.432723 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260804 14:53:16.434350 32172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:16.442124 32172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e8182d4ea604f7b9ee1d68918f525d8: Bootstrap starting.
I20260804 14:53:16.445021 32172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e8182d4ea604f7b9ee1d68918f525d8: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:16.446076 32172 log.cc:824] T 00000000000000000000000000000000 P 0e8182d4ea604f7b9ee1d68918f525d8: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:16.448897 32172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e8182d4ea604f7b9ee1d68918f525d8: No bootstrap required, opened a new log
I20260804 14:53:16.453702 32172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e8182d4ea604f7b9ee1d68918f525d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e8182d4ea604f7b9ee1d68918f525d8" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 40781 } }
I20260804 14:53:16.454073 32172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e8182d4ea604f7b9ee1d68918f525d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:53:16.454175 32172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e8182d4ea604f7b9ee1d68918f525d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e8182d4ea604f7b9ee1d68918f525d8, State: Initialized, Role: FOLLOWER
I20260804 14:53:16.454747 32172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e8182d4ea604f7b9ee1d68918f525d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e8182d4ea604f7b9ee1d68918f525d8" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 40781 } }
I20260804 14:53:16.454948 32172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e8182d4ea604f7b9ee1d68918f525d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260804 14:53:16.455053 32172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e8182d4ea604f7b9ee1d68918f525d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260804 14:53:16.455224 32172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e8182d4ea604f7b9ee1d68918f525d8 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:16.456503 32172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e8182d4ea604f7b9ee1d68918f525d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e8182d4ea604f7b9ee1d68918f525d8" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 40781 } }
I20260804 14:53:16.457013 32172 leader_election.cc:302] T 00000000000000000000000000000000 P 0e8182d4ea604f7b9ee1d68918f525d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e8182d4ea604f7b9ee1d68918f525d8; no voters:
I20260804 14:53:16.457468 32172 leader_election.cc:288] T 00000000000000000000000000000000 P 0e8182d4ea604f7b9ee1d68918f525d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260804 14:53:16.457602 32178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e8182d4ea604f7b9ee1d68918f525d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:16.457813 32178 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e8182d4ea604f7b9ee1d68918f525d8 [term 1 LEADER]: Becoming Leader. State: Replica: 0e8182d4ea604f7b9ee1d68918f525d8, State: Running, Role: LEADER
I20260804 14:53:16.458264 32178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e8182d4ea604f7b9ee1d68918f525d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e8182d4ea604f7b9ee1d68918f525d8" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 40781 } }
I20260804 14:53:16.459161 32172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e8182d4ea604f7b9ee1d68918f525d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260804 14:53:16.460246 32180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e8182d4ea604f7b9ee1d68918f525d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e8182d4ea604f7b9ee1d68918f525d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e8182d4ea604f7b9ee1d68918f525d8" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 40781 } } }
I20260804 14:53:16.460441 32180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e8182d4ea604f7b9ee1d68918f525d8 [sys.catalog]: This master's current role is: LEADER
I20260804 14:53:16.460739 32181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e8182d4ea604f7b9ee1d68918f525d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e8182d4ea604f7b9ee1d68918f525d8. Latest consensus state: current_term: 1 leader_uuid: "0e8182d4ea604f7b9ee1d68918f525d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e8182d4ea604f7b9ee1d68918f525d8" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 40781 } } }
I20260804 14:53:16.460842 32181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e8182d4ea604f7b9ee1d68918f525d8 [sys.catalog]: This master's current role is: LEADER
I20260804 14:53:16.460906 32194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260804 14:53:16.464380 32194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260804 14:53:16.470983 32194 catalog_manager.cc:1347] Generated new cluster ID: e8a93c426bdd4de6b88b0fb640ce5155
I20260804 14:53:16.471138 32194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260804 14:53:16.479135 32194 catalog_manager.cc:1370] Generated new certificate authority record
I20260804 14:53:16.480229 32194 catalog_manager.cc:1504] Loading token signing keys...
I20260804 14:53:16.495182 32194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e8182d4ea604f7b9ee1d68918f525d8: Generated new TSK 0
I20260804 14:53:16.496049 32194 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260804 14:53:16.500255 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.129:0
--local_ip_for_outbound_sockets=127.24.6.129
--webserver_interface=127.24.6.129
--webserver_port=0
--tserver_master_addrs=127.24.6.190:40781
--builtin_ntp_servers=127.24.6.148:37271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20260804 14:53:16.627928 32208 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260804 14:53:16.628257 32208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:16.628278 32208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:16.628311 32208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:16.632476 32208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:16.632586 32208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.129
I20260804 14:53:16.637648 32208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:37271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.24.6.129
--webserver_port=0
--tserver_master_addrs=127.24.6.190:40781
--never_fsync=true
--heap_profile_path=/tmp/kudu.32208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.129
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:16.638873 32208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:16.640377 32208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:16.651062 32216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:16.651072 32217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:16.651115 32219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:16.651530 32208 server_base.cc:1034] running on GCE node
I20260804 14:53:16.652138 32208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:16.652954 32208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:16.654181 32208 hybrid_clock.cc:648] HybridClock initialized: now 1785855196654139 us; error 55 us; skew 500 ppm
I20260804 14:53:16.656517 32208 webserver.cc:460] Webserver started at http://127.24.6.129:37747/ using document root <none> and password file <none>
I20260804 14:53:16.657262 32208 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:16.657380 32208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:16.657657 32208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:16.660080 32208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "417f5a60b3a1419eb1600c18349cff7c"
format_stamp: "Formatted at 2026-08-04 14:53:16 on dist-test-slave-92m1"
I20260804 14:53:16.660745 32208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "417f5a60b3a1419eb1600c18349cff7c"
format_stamp: "Formatted at 2026-08-04 14:53:16 on dist-test-slave-92m1"
I20260804 14:53:16.664988 32208 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260804 14:53:16.668020 32227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:16.669487 32208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260804 14:53:16.669658 32208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "417f5a60b3a1419eb1600c18349cff7c"
format_stamp: "Formatted at 2026-08-04 14:53:16 on dist-test-slave-92m1"
I20260804 14:53:16.669801 32208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:16.681231 32208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:16.682054 32208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:16.682297 32208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:16.682982 32208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:53:16.684191 32208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:53:16.684273 32208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:16.684353 32208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:53:16.684399 32208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:16.694983 32208 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.129:38505
I20260804 14:53:16.695042 32398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.129:38505 every 8 connection(s)
I20260804 14:53:16.696192 32208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260804 14:53:16.699050 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 32208
I20260804 14:53:16.699239 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260804 14:53:16.701963 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.130:0
--local_ip_for_outbound_sockets=127.24.6.130
--webserver_interface=127.24.6.130
--webserver_port=0
--tserver_master_addrs=127.24.6.190:40781
--builtin_ntp_servers=127.24.6.148:37271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260804 14:53:16.709625 32399 heartbeater.cc:347] Connected to a master server at 127.24.6.190:40781
I20260804 14:53:16.709942 32399 heartbeater.cc:464] Registering TS with master...
I20260804 14:53:16.710609 32399 heartbeater.cc:511] Master 127.24.6.190:40781 requested a full tablet report, sending...
I20260804 14:53:16.712757 32117 ts_manager.cc:194] Registered new tserver with Master: 417f5a60b3a1419eb1600c18349cff7c (127.24.6.129:38505)
I20260804 14:53:16.714746 32117 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.129:60899
W20260804 14:53:16.829696 32402 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260804 14:53:16.830034 32402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:16.830056 32402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:16.830090 32402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:16.834353 32402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:16.834479 32402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.130
I20260804 14:53:16.839608 32402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:37271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.24.6.130
--webserver_port=0
--tserver_master_addrs=127.24.6.190:40781
--never_fsync=true
--heap_profile_path=/tmp/kudu.32402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.130
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:16.840848 32402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:16.842454 32402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:16.854041 32414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:16.854044 32416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:16.854044 32413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:16.854437 32402 server_base.cc:1034] running on GCE node
I20260804 14:53:16.855178 32402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:16.856082 32402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:16.857291 32402 hybrid_clock.cc:648] HybridClock initialized: now 1785855196857276 us; error 42 us; skew 500 ppm
I20260804 14:53:16.859934 32402 webserver.cc:460] Webserver started at http://127.24.6.130:43367/ using document root <none> and password file <none>
I20260804 14:53:16.860867 32402 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:16.860993 32402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:16.861295 32402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:16.863824 32402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "0b7ef1df2b4f4238b113c5b663575d1e"
format_stamp: "Formatted at 2026-08-04 14:53:16 on dist-test-slave-92m1"
I20260804 14:53:16.864552 32402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "0b7ef1df2b4f4238b113c5b663575d1e"
format_stamp: "Formatted at 2026-08-04 14:53:16 on dist-test-slave-92m1"
I20260804 14:53:16.869391 32402 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260804 14:53:16.872778 32424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:16.874445 32402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260804 14:53:16.874651 32402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "0b7ef1df2b4f4238b113c5b663575d1e"
format_stamp: "Formatted at 2026-08-04 14:53:16 on dist-test-slave-92m1"
I20260804 14:53:16.874835 32402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:16.889686 32402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:16.890630 32402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:16.890873 32402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:16.891645 32402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:53:16.892882 32402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:53:16.892966 32402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:16.893064 32402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:53:16.893107 32402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:16.903249 32402 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.130:45677
I20260804 14:53:16.903268 32597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.130:45677 every 8 connection(s)
I20260804 14:53:16.904276 32402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260804 14:53:16.912590 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 32402
I20260804 14:53:16.912799 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260804 14:53:16.915865 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.131:0
--local_ip_for_outbound_sockets=127.24.6.131
--webserver_interface=127.24.6.131
--webserver_port=0
--tserver_master_addrs=127.24.6.190:40781
--builtin_ntp_servers=127.24.6.148:37271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260804 14:53:16.917025 32598 heartbeater.cc:347] Connected to a master server at 127.24.6.190:40781
I20260804 14:53:16.917357 32598 heartbeater.cc:464] Registering TS with master...
I20260804 14:53:16.918141 32598 heartbeater.cc:511] Master 127.24.6.190:40781 requested a full tablet report, sending...
I20260804 14:53:16.919744 32117 ts_manager.cc:194] Registered new tserver with Master: 0b7ef1df2b4f4238b113c5b663575d1e (127.24.6.130:45677)
I20260804 14:53:16.920500 32117 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.130:33249
W20260804 14:53:17.050247 32605 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260804 14:53:17.050544 32605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:17.050565 32605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:17.050645 32605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:17.055104 32605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:17.055325 32605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.131
I20260804 14:53:17.060925 32605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:37271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.24.6.131
--webserver_port=0
--tserver_master_addrs=127.24.6.190:40781
--never_fsync=true
--heap_profile_path=/tmp/kudu.32605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.131
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:17.062233 32605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:17.063884 32605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:17.075090 32623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:17.075129 32621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:17.075152 32605 server_base.cc:1034] running on GCE node
W20260804 14:53:17.075057 32620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:17.075907 32605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:17.076602 32605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:17.077775 32605 hybrid_clock.cc:648] HybridClock initialized: now 1785855197077758 us; error 38 us; skew 500 ppm
I20260804 14:53:17.080148 32605 webserver.cc:460] Webserver started at http://127.24.6.131:38243/ using document root <none> and password file <none>
I20260804 14:53:17.080838 32605 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:17.080914 32605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:17.081161 32605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:17.083498 32605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "9b925b2fe37a42988d2cd89c05f5689b"
format_stamp: "Formatted at 2026-08-04 14:53:17 on dist-test-slave-92m1"
I20260804 14:53:17.084168 32605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "9b925b2fe37a42988d2cd89c05f5689b"
format_stamp: "Formatted at 2026-08-04 14:53:17 on dist-test-slave-92m1"
I20260804 14:53:17.089016 32605 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260804 14:53:17.092810 32630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:17.094741 32605 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260804 14:53:17.094956 32605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9b925b2fe37a42988d2cd89c05f5689b"
format_stamp: "Formatted at 2026-08-04 14:53:17 on dist-test-slave-92m1"
I20260804 14:53:17.095165 32605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:17.128340 32605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:17.129235 32605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:17.129483 32605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:17.130226 32605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:53:17.131448 32605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:53:17.131505 32605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:17.131556 32605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:53:17.131572 32605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:17.142341 32605 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.131:41791
I20260804 14:53:17.142403 324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.131:41791 every 8 connection(s)
I20260804 14:53:17.143635 32605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260804 14:53:17.146241 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 32605
I20260804 14:53:17.146384 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260804 14:53:17.156754 325 heartbeater.cc:347] Connected to a master server at 127.24.6.190:40781
I20260804 14:53:17.157047 325 heartbeater.cc:464] Registering TS with master...
I20260804 14:53:17.157658 325 heartbeater.cc:511] Master 127.24.6.190:40781 requested a full tablet report, sending...
I20260804 14:53:17.159032 32117 ts_manager.cc:194] Registered new tserver with Master: 9b925b2fe37a42988d2cd89c05f5689b (127.24.6.131:41791)
I20260804 14:53:17.160037 32117 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.131:37819
I20260804 14:53:17.161562 24602 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260804 14:53:17.173060 32117 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49146:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20260804 14:53:17.176582 32117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260804 14:53:17.196593 32306 tablet_service.cc:1467] Processing CreateTablet for tablet b7ae0618b88a4ea09e254d4c1b6794e2 (DEFAULT_TABLE table=TestTable [id=7f1a1cca8851494db2d1d118857f6510]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:53:17.196602 32699 tablet_service.cc:1467] Processing CreateTablet for tablet b7ae0618b88a4ea09e254d4c1b6794e2 (DEFAULT_TABLE table=TestTable [id=7f1a1cca8851494db2d1d118857f6510]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:53:17.198006 32306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7ae0618b88a4ea09e254d4c1b6794e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:17.198006 32699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7ae0618b88a4ea09e254d4c1b6794e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:17.197688 32494 tablet_service.cc:1467] Processing CreateTablet for tablet b7ae0618b88a4ea09e254d4c1b6794e2 (DEFAULT_TABLE table=TestTable [id=7f1a1cca8851494db2d1d118857f6510]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:53:17.198666 32494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7ae0618b88a4ea09e254d4c1b6794e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:17.205729 347 tablet_bootstrap.cc:492] T b7ae0618b88a4ea09e254d4c1b6794e2 P 0b7ef1df2b4f4238b113c5b663575d1e: Bootstrap starting.
I20260804 14:53:17.207005 348 tablet_bootstrap.cc:492] T b7ae0618b88a4ea09e254d4c1b6794e2 P 417f5a60b3a1419eb1600c18349cff7c: Bootstrap starting.
I20260804 14:53:17.207585 350 tablet_bootstrap.cc:492] T b7ae0618b88a4ea09e254d4c1b6794e2 P 9b925b2fe37a42988d2cd89c05f5689b: Bootstrap starting.
I20260804 14:53:17.208629 347 tablet_bootstrap.cc:654] T b7ae0618b88a4ea09e254d4c1b6794e2 P 0b7ef1df2b4f4238b113c5b663575d1e: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:17.209395 347 log.cc:824] T b7ae0618b88a4ea09e254d4c1b6794e2 P 0b7ef1df2b4f4238b113c5b663575d1e: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:17.209774 348 tablet_bootstrap.cc:654] T b7ae0618b88a4ea09e254d4c1b6794e2 P 417f5a60b3a1419eb1600c18349cff7c: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:17.210040 350 tablet_bootstrap.cc:654] T b7ae0618b88a4ea09e254d4c1b6794e2 P 9b925b2fe37a42988d2cd89c05f5689b: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:17.210825 348 log.cc:824] T b7ae0618b88a4ea09e254d4c1b6794e2 P 417f5a60b3a1419eb1600c18349cff7c: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:17.210896 350 log.cc:824] T b7ae0618b88a4ea09e254d4c1b6794e2 P 9b925b2fe37a42988d2cd89c05f5689b: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:17.211753 347 tablet_bootstrap.cc:492] T b7ae0618b88a4ea09e254d4c1b6794e2 P 0b7ef1df2b4f4238b113c5b663575d1e: No bootstrap required, opened a new log
I20260804 14:53:17.211985 347 ts_tablet_manager.cc:1397] T b7ae0618b88a4ea09e254d4c1b6794e2 P 0b7ef1df2b4f4238b113c5b663575d1e: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260804 14:53:17.213124 348 tablet_bootstrap.cc:492] T b7ae0618b88a4ea09e254d4c1b6794e2 P 417f5a60b3a1419eb1600c18349cff7c: No bootstrap required, opened a new log
I20260804 14:53:17.213124 350 tablet_bootstrap.cc:492] T b7ae0618b88a4ea09e254d4c1b6794e2 P 9b925b2fe37a42988d2cd89c05f5689b: No bootstrap required, opened a new log
I20260804 14:53:17.213346 350 ts_tablet_manager.cc:1397] T b7ae0618b88a4ea09e254d4c1b6794e2 P 9b925b2fe37a42988d2cd89c05f5689b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260804 14:53:17.213346 348 ts_tablet_manager.cc:1397] T b7ae0618b88a4ea09e254d4c1b6794e2 P 417f5a60b3a1419eb1600c18349cff7c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260804 14:53:17.215878 347 raft_consensus.cc:348] T b7ae0618b88a4ea09e254d4c1b6794e2 P 0b7ef1df2b4f4238b113c5b663575d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7ef1df2b4f4238b113c5b663575d1e" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 45677 } } peers { permanent_uuid: "417f5a60b3a1419eb1600c18349cff7c" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38505 } } peers { permanent_uuid: "9b925b2fe37a42988d2cd89c05f5689b" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41791 } }
I20260804 14:53:17.216178 347 raft_consensus.cc:729] T b7ae0618b88a4ea09e254d4c1b6794e2 P 0b7ef1df2b4f4238b113c5b663575d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b7ef1df2b4f4238b113c5b663575d1e, State: Initialized, Role: FOLLOWER
I20260804 14:53:17.216498 347 consensus_queue.cc:260] T b7ae0618b88a4ea09e254d4c1b6794e2 P 0b7ef1df2b4f4238b113c5b663575d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7ef1df2b4f4238b113c5b663575d1e" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 45677 } } peers { permanent_uuid: "417f5a60b3a1419eb1600c18349cff7c" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38505 } } peers { permanent_uuid: "9b925b2fe37a42988d2cd89c05f5689b" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41791 } }
I20260804 14:53:17.217273 32598 heartbeater.cc:503] Master 127.24.6.190:40781 was elected leader, sending a full tablet report...
I20260804 14:53:17.217744 347 ts_tablet_manager.cc:1428] T b7ae0618b88a4ea09e254d4c1b6794e2 P 0b7ef1df2b4f4238b113c5b663575d1e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260804 14:53:17.218743 348 raft_consensus.cc:348] T b7ae0618b88a4ea09e254d4c1b6794e2 P 417f5a60b3a1419eb1600c18349cff7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7ef1df2b4f4238b113c5b663575d1e" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 45677 } } peers { permanent_uuid: "417f5a60b3a1419eb1600c18349cff7c" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38505 } } peers { permanent_uuid: "9b925b2fe37a42988d2cd89c05f5689b" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41791 } }
I20260804 14:53:17.219084 348 raft_consensus.cc:729] T b7ae0618b88a4ea09e254d4c1b6794e2 P 417f5a60b3a1419eb1600c18349cff7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417f5a60b3a1419eb1600c18349cff7c, State: Initialized, Role: FOLLOWER
I20260804 14:53:17.218906 350 raft_consensus.cc:348] T b7ae0618b88a4ea09e254d4c1b6794e2 P 9b925b2fe37a42988d2cd89c05f5689b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7ef1df2b4f4238b113c5b663575d1e" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 45677 } } peers { permanent_uuid: "417f5a60b3a1419eb1600c18349cff7c" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38505 } } peers { permanent_uuid: "9b925b2fe37a42988d2cd89c05f5689b" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41791 } }
I20260804 14:53:17.219173 350 raft_consensus.cc:729] T b7ae0618b88a4ea09e254d4c1b6794e2 P 9b925b2fe37a42988d2cd89c05f5689b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b925b2fe37a42988d2cd89c05f5689b, State: Initialized, Role: FOLLOWER
I20260804 14:53:17.219506 350 consensus_queue.cc:260] T b7ae0618b88a4ea09e254d4c1b6794e2 P 9b925b2fe37a42988d2cd89c05f5689b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7ef1df2b4f4238b113c5b663575d1e" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 45677 } } peers { permanent_uuid: "417f5a60b3a1419eb1600c18349cff7c" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38505 } } peers { permanent_uuid: "9b925b2fe37a42988d2cd89c05f5689b" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41791 } }
I20260804 14:53:17.219506 348 consensus_queue.cc:260] T b7ae0618b88a4ea09e254d4c1b6794e2 P 417f5a60b3a1419eb1600c18349cff7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7ef1df2b4f4238b113c5b663575d1e" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 45677 } } peers { permanent_uuid: "417f5a60b3a1419eb1600c18349cff7c" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38505 } } peers { permanent_uuid: "9b925b2fe37a42988d2cd89c05f5689b" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41791 } }
I20260804 14:53:17.220568 325 heartbeater.cc:503] Master 127.24.6.190:40781 was elected leader, sending a full tablet report...
I20260804 14:53:17.220772 348 ts_tablet_manager.cc:1428] T b7ae0618b88a4ea09e254d4c1b6794e2 P 417f5a60b3a1419eb1600c18349cff7c: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260804 14:53:17.221601 350 ts_tablet_manager.cc:1428] T b7ae0618b88a4ea09e254d4c1b6794e2 P 9b925b2fe37a42988d2cd89c05f5689b: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260804 14:53:17.221877 32399 heartbeater.cc:503] Master 127.24.6.190:40781 was elected leader, sending a full tablet report...
I20260804 14:53:17.232462 24602 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260804 14:53:17.234208 24602 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 417f5a60b3a1419eb1600c18349cff7c to finish bootstrapping
I20260804 14:53:17.241737 24602 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0b7ef1df2b4f4238b113c5b663575d1e to finish bootstrapping
I20260804 14:53:17.249117 24602 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 9b925b2fe37a42988d2cd89c05f5689b to finish bootstrapping
I20260804 14:53:17.269146 32524 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b7ae0618b88a4ea09e254d4c1b6794e2"
dest_uuid: "0b7ef1df2b4f4238b113c5b663575d1e"
from {username='slave'} at 127.0.0.1:49044
I20260804 14:53:17.269459 32524 raft_consensus.cc:482] T b7ae0618b88a4ea09e254d4c1b6794e2 P 0b7ef1df2b4f4238b113c5b663575d1e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260804 14:53:17.269631 32524 raft_consensus.cc:3037] T b7ae0618b88a4ea09e254d4c1b6794e2 P 0b7ef1df2b4f4238b113c5b663575d1e [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:17.271240 32524 raft_consensus.cc:504] T b7ae0618b88a4ea09e254d4c1b6794e2 P 0b7ef1df2b4f4238b113c5b663575d1e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7ef1df2b4f4238b113c5b663575d1e" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 45677 } } peers { permanent_uuid: "417f5a60b3a1419eb1600c18349cff7c" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38505 } } peers { permanent_uuid: "9b925b2fe37a42988d2cd89c05f5689b" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41791 } }
I20260804 14:53:17.272790 32524 leader_election.cc:288] T b7ae0618b88a4ea09e254d4c1b6794e2 P 0b7ef1df2b4f4238b113c5b663575d1e [CANDIDATE]: Term 1 election: Requested vote from peers 417f5a60b3a1419eb1600c18349cff7c (127.24.6.129:38505), 9b925b2fe37a42988d2cd89c05f5689b (127.24.6.131:41791)
I20260804 14:53:17.279083 32725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7ae0618b88a4ea09e254d4c1b6794e2" candidate_uuid: "0b7ef1df2b4f4238b113c5b663575d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9b925b2fe37a42988d2cd89c05f5689b"
I20260804 14:53:17.279407 32725 raft_consensus.cc:3037] T b7ae0618b88a4ea09e254d4c1b6794e2 P 9b925b2fe37a42988d2cd89c05f5689b [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:17.280543 32332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7ae0618b88a4ea09e254d4c1b6794e2" candidate_uuid: "0b7ef1df2b4f4238b113c5b663575d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "417f5a60b3a1419eb1600c18349cff7c"
I20260804 14:53:17.280862 32332 raft_consensus.cc:3037] T b7ae0618b88a4ea09e254d4c1b6794e2 P 417f5a60b3a1419eb1600c18349cff7c [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:17.280980 32725 raft_consensus.cc:2454] T b7ae0618b88a4ea09e254d4c1b6794e2 P 9b925b2fe37a42988d2cd89c05f5689b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b7ef1df2b4f4238b113c5b663575d1e in term 1.
I20260804 14:53:17.281517 32426 leader_election.cc:302] T b7ae0618b88a4ea09e254d4c1b6794e2 P 0b7ef1df2b4f4238b113c5b663575d1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b7ef1df2b4f4238b113c5b663575d1e, 9b925b2fe37a42988d2cd89c05f5689b; no voters:
I20260804 14:53:17.281860 355 raft_consensus.cc:2781] T b7ae0618b88a4ea09e254d4c1b6794e2 P 0b7ef1df2b4f4238b113c5b663575d1e [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:17.282755 32332 raft_consensus.cc:2454] T b7ae0618b88a4ea09e254d4c1b6794e2 P 417f5a60b3a1419eb1600c18349cff7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b7ef1df2b4f4238b113c5b663575d1e in term 1.
I20260804 14:53:17.283653 355 raft_consensus.cc:686] T b7ae0618b88a4ea09e254d4c1b6794e2 P 0b7ef1df2b4f4238b113c5b663575d1e [term 1 LEADER]: Becoming Leader. State: Replica: 0b7ef1df2b4f4238b113c5b663575d1e, State: Running, Role: LEADER
I20260804 14:53:17.284029 355 consensus_queue.cc:237] T b7ae0618b88a4ea09e254d4c1b6794e2 P 0b7ef1df2b4f4238b113c5b663575d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7ef1df2b4f4238b113c5b663575d1e" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 45677 } } peers { permanent_uuid: "417f5a60b3a1419eb1600c18349cff7c" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38505 } } peers { permanent_uuid: "9b925b2fe37a42988d2cd89c05f5689b" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41791 } }
I20260804 14:53:17.288736 32116 catalog_manager.cc:5495] T b7ae0618b88a4ea09e254d4c1b6794e2 P 0b7ef1df2b4f4238b113c5b663575d1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b7ef1df2b4f4238b113c5b663575d1e (127.24.6.130). New cstate: current_term: 1 leader_uuid: "0b7ef1df2b4f4238b113c5b663575d1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7ef1df2b4f4238b113c5b663575d1e" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 45677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "417f5a60b3a1419eb1600c18349cff7c" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b925b2fe37a42988d2cd89c05f5689b" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41791 } health_report { overall_health: UNKNOWN } } }
I20260804 14:53:17.296362 32725 raft_consensus.cc:1261] T b7ae0618b88a4ea09e254d4c1b6794e2 P 9b925b2fe37a42988d2cd89c05f5689b [term 1 FOLLOWER]: Refusing update from remote peer 0b7ef1df2b4f4238b113c5b663575d1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260804 14:53:17.296844 32332 raft_consensus.cc:1261] T b7ae0618b88a4ea09e254d4c1b6794e2 P 417f5a60b3a1419eb1600c18349cff7c [term 1 FOLLOWER]: Refusing update from remote peer 0b7ef1df2b4f4238b113c5b663575d1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260804 14:53:17.297138 355 consensus_queue.cc:1035] T b7ae0618b88a4ea09e254d4c1b6794e2 P 0b7ef1df2b4f4238b113c5b663575d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b925b2fe37a42988d2cd89c05f5689b" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260804 14:53:17.297617 363 consensus_queue.cc:1035] T b7ae0618b88a4ea09e254d4c1b6794e2 P 0b7ef1df2b4f4238b113c5b663575d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "417f5a60b3a1419eb1600c18349cff7c" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260804 14:53:17.305109 367 mvcc.cc:205] Tried to move back new op lower bound from 7314862888116666368 to 7314862888079441920. Current Snapshot: MvccSnapshot[applied={T|T < 7314862888116666368}]
I20260804 14:53:17.306416 371 mvcc.cc:205] Tried to move back new op lower bound from 7314862888116666368 to 7314862888079441920. Current Snapshot: MvccSnapshot[applied={T|T < 7314862888116666368}]
I20260804 14:53:17.328540 32524 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b7ae0618b88a4ea09e254d4c1b6794e2"
dest_uuid: "0b7ef1df2b4f4238b113c5b663575d1e"
from {username='slave'} at 127.0.0.1:49044
I20260804 14:53:17.328889 32524 raft_consensus.cc:580] T b7ae0618b88a4ea09e254d4c1b6794e2 P 0b7ef1df2b4f4238b113c5b663575d1e [term 1 LEADER]: Received request to step down
I20260804 14:53:17.328972 32524 raft_consensus.cc:3032] T b7ae0618b88a4ea09e254d4c1b6794e2 P 0b7ef1df2b4f4238b113c5b663575d1e [term 1 LEADER]: Stepping down as leader of term 1
I20260804 14:53:17.329018 32524 raft_consensus.cc:729] T b7ae0618b88a4ea09e254d4c1b6794e2 P 0b7ef1df2b4f4238b113c5b663575d1e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0b7ef1df2b4f4238b113c5b663575d1e, State: Running, Role: LEADER
I20260804 14:53:17.329172 32524 consensus_queue.cc:260] T b7ae0618b88a4ea09e254d4c1b6794e2 P 0b7ef1df2b4f4238b113c5b663575d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7ef1df2b4f4238b113c5b663575d1e" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 45677 } } peers { permanent_uuid: "417f5a60b3a1419eb1600c18349cff7c" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 38505 } } peers { permanent_uuid: "9b925b2fe37a42988d2cd89c05f5689b" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41791 } }
I20260804 14:53:17.329504 32524 raft_consensus.cc:3037] T b7ae0618b88a4ea09e254d4c1b6794e2 P 0b7ef1df2b4f4238b113c5b663575d1e [term 1 FOLLOWER]: Advancing to term 2
I20260804 14:53:17.331346 32524 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b7ae0618b88a4ea09e254d4c1b6794e2"
dest_uuid: "0b7ef1df2b4f4238b113c5b663575d1e"
from {username='slave'} at 127.0.0.1:49044
I20260804 14:53:17.331506 32524 raft_consensus.cc:573] T b7ae0618b88a4ea09e254d4c1b6794e2 P 0b7ef1df2b4f4238b113c5b663575d1e [term 2 FOLLOWER]: Rejecting request to step down while not leader
W20260804 14:53:17.333199 32467 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49044: Illegal state: replica 0b7ef1df2b4f4238b113c5b663575d1e is not leader of this config: current role FOLLOWER
I20260804 14:53:17.334184 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 32208
I20260804 14:53:17.346438 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 32402
I20260804 14:53:17.360709 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 32605
I20260804 14:53:17.377300 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 32070
2026-08-04T14:53:17Z chronyd exiting
[ OK ] RaftConsensusElectionITest.LeaderStepDown (1190 ms)
[ RUN ] RaftConsensusElectionITest.StepDownWithSlowFollower
I20260804 14:53:17.410988 24602 test_util.cc:268] Using random seed: -794200411
I20260804 14:53:17.411943 24602 ts_itest-base.cc:112] Starting cluster with:
I20260804 14:53:17.411990 24602 ts_itest-base.cc:113] --------------
I20260804 14:53:17.412003 24602 ts_itest-base.cc:114] 3 tablet servers
I20260804 14:53:17.412014 24602 ts_itest-base.cc:115] 3 replicas per TS
I20260804 14:53:17.412025 24602 ts_itest-base.cc:116] --------------
2026-08-04T14:53:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-04T14:53:17Z Disabled control of system clock
I20260804 14:53:17.424317 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.6.190:46077
--webserver_interface=127.24.6.190
--webserver_port=0
--builtin_ntp_servers=127.24.6.148:34955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.6.190:46077
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20260804 14:53:17.553503 384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:17.554004 384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:17.554097 384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:17.559935 384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260804 14:53:17.560060 384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:17.560096 384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260804 14:53:17.560142 384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260804 14:53:17.566200 384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:34955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.24.6.190:46077
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.6.190:46077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.24.6.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:17.567648 384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:17.569249 384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:17.579286 391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:17.579244 390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:17.579265 393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:17.579872 384 server_base.cc:1034] running on GCE node
I20260804 14:53:17.580626 384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:17.581933 384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:17.583130 384 hybrid_clock.cc:648] HybridClock initialized: now 1785855197583112 us; error 39 us; skew 500 ppm
I20260804 14:53:17.585528 384 webserver.cc:460] Webserver started at http://127.24.6.190:44177/ using document root <none> and password file <none>
I20260804 14:53:17.586205 384 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:17.586277 384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:17.586537 384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:17.588869 384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f5732aa4daa24bcfbccb42e3f6278f73"
format_stamp: "Formatted at 2026-08-04 14:53:17 on dist-test-slave-92m1"
I20260804 14:53:17.589599 384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f5732aa4daa24bcfbccb42e3f6278f73"
format_stamp: "Formatted at 2026-08-04 14:53:17 on dist-test-slave-92m1"
I20260804 14:53:17.595033 384 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260804 14:53:17.598037 400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:17.599712 384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260804 14:53:17.599910 384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f5732aa4daa24bcfbccb42e3f6278f73"
format_stamp: "Formatted at 2026-08-04 14:53:17 on dist-test-slave-92m1"
I20260804 14:53:17.600059 384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:17.617441 384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:17.618285 384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:17.618533 384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:17.627517 384 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.190:46077
I20260804 14:53:17.627544 476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.190:46077 every 8 connection(s)
I20260804 14:53:17.628800 384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260804 14:53:17.632635 477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:17.632683 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 384
I20260804 14:53:17.632824 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260804 14:53:17.638815 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5732aa4daa24bcfbccb42e3f6278f73: Bootstrap starting.
I20260804 14:53:17.641719 477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5732aa4daa24bcfbccb42e3f6278f73: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:17.642791 477 log.cc:824] T 00000000000000000000000000000000 P f5732aa4daa24bcfbccb42e3f6278f73: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:17.645196 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5732aa4daa24bcfbccb42e3f6278f73: No bootstrap required, opened a new log
I20260804 14:53:17.649271 477 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5732aa4daa24bcfbccb42e3f6278f73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5732aa4daa24bcfbccb42e3f6278f73" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 46077 } }
I20260804 14:53:17.649652 477 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5732aa4daa24bcfbccb42e3f6278f73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:53:17.649698 477 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5732aa4daa24bcfbccb42e3f6278f73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5732aa4daa24bcfbccb42e3f6278f73, State: Initialized, Role: FOLLOWER
I20260804 14:53:17.650117 477 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5732aa4daa24bcfbccb42e3f6278f73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5732aa4daa24bcfbccb42e3f6278f73" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 46077 } }
I20260804 14:53:17.650246 477 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5732aa4daa24bcfbccb42e3f6278f73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260804 14:53:17.650291 477 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5732aa4daa24bcfbccb42e3f6278f73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260804 14:53:17.650363 477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5732aa4daa24bcfbccb42e3f6278f73 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:17.651448 477 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5732aa4daa24bcfbccb42e3f6278f73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5732aa4daa24bcfbccb42e3f6278f73" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 46077 } }
I20260804 14:53:17.651813 477 leader_election.cc:302] T 00000000000000000000000000000000 P f5732aa4daa24bcfbccb42e3f6278f73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5732aa4daa24bcfbccb42e3f6278f73; no voters:
I20260804 14:53:17.652225 477 leader_election.cc:288] T 00000000000000000000000000000000 P f5732aa4daa24bcfbccb42e3f6278f73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260804 14:53:17.652354 486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5732aa4daa24bcfbccb42e3f6278f73 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:17.652622 486 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5732aa4daa24bcfbccb42e3f6278f73 [term 1 LEADER]: Becoming Leader. State: Replica: f5732aa4daa24bcfbccb42e3f6278f73, State: Running, Role: LEADER
I20260804 14:53:17.653075 486 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5732aa4daa24bcfbccb42e3f6278f73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5732aa4daa24bcfbccb42e3f6278f73" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 46077 } }
I20260804 14:53:17.653748 477 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5732aa4daa24bcfbccb42e3f6278f73 [sys.catalog]: configured and running, proceeding with master startup.
I20260804 14:53:17.655411 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5732aa4daa24bcfbccb42e3f6278f73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5732aa4daa24bcfbccb42e3f6278f73. Latest consensus state: current_term: 1 leader_uuid: "f5732aa4daa24bcfbccb42e3f6278f73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5732aa4daa24bcfbccb42e3f6278f73" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 46077 } } }
I20260804 14:53:17.655583 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5732aa4daa24bcfbccb42e3f6278f73 [sys.catalog]: This master's current role is: LEADER
I20260804 14:53:17.656010 488 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5732aa4daa24bcfbccb42e3f6278f73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5732aa4daa24bcfbccb42e3f6278f73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5732aa4daa24bcfbccb42e3f6278f73" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 46077 } } }
I20260804 14:53:17.656134 488 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5732aa4daa24bcfbccb42e3f6278f73 [sys.catalog]: This master's current role is: LEADER
I20260804 14:53:17.656085 499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260804 14:53:17.660316 499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260804 14:53:17.668540 499 catalog_manager.cc:1347] Generated new cluster ID: 126f046030b049ccb4f7235de1402e73
I20260804 14:53:17.668767 499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260804 14:53:17.691851 499 catalog_manager.cc:1370] Generated new certificate authority record
I20260804 14:53:17.693202 499 catalog_manager.cc:1504] Loading token signing keys...
I20260804 14:53:17.704137 499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5732aa4daa24bcfbccb42e3f6278f73: Generated new TSK 0
I20260804 14:53:17.705277 499 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260804 14:53:17.719039 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.129:0
--local_ip_for_outbound_sockets=127.24.6.129
--webserver_interface=127.24.6.129
--webserver_port=0
--tserver_master_addrs=127.24.6.190:46077
--builtin_ntp_servers=127.24.6.148:34955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--consensus_rpc_timeout_ms=15000 with env {}
W20260804 14:53:17.852310 511 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260804 14:53:17.852694 511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:17.852717 511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:17.852754 511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:17.857462 511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:17.857591 511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.129
I20260804 14:53:17.862914 511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:34955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=15000
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.24.6.129
--webserver_port=0
--tserver_master_addrs=127.24.6.190:46077
--never_fsync=true
--heap_profile_path=/tmp/kudu.511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.129
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:17.864274 511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:17.865756 511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:17.877106 522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:17.877108 520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:17.877108 519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:17.877152 511 server_base.cc:1034] running on GCE node
I20260804 14:53:17.877871 511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:17.878616 511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:17.879812 511 hybrid_clock.cc:648] HybridClock initialized: now 1785855197879788 us; error 37 us; skew 500 ppm
I20260804 14:53:17.882036 511 webserver.cc:460] Webserver started at http://127.24.6.129:37721/ using document root <none> and password file <none>
I20260804 14:53:17.882721 511 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:17.882797 511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:17.883100 511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:17.885316 511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "bc3e61f1d2fb482f9d6490c298efa2f0"
format_stamp: "Formatted at 2026-08-04 14:53:17 on dist-test-slave-92m1"
I20260804 14:53:17.885942 511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "bc3e61f1d2fb482f9d6490c298efa2f0"
format_stamp: "Formatted at 2026-08-04 14:53:17 on dist-test-slave-92m1"
I20260804 14:53:17.890530 511 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260804 14:53:17.893599 531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:17.895336 511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260804 14:53:17.895515 511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "bc3e61f1d2fb482f9d6490c298efa2f0"
format_stamp: "Formatted at 2026-08-04 14:53:17 on dist-test-slave-92m1"
I20260804 14:53:17.895673 511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:17.918305 511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:17.919118 511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:17.919374 511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:17.920028 511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:53:17.921084 511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:53:17.921160 511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:17.921231 511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:53:17.921281 511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:17.931198 511 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.129:40657
I20260804 14:53:17.931248 698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.129:40657 every 8 connection(s)
I20260804 14:53:17.932268 511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260804 14:53:17.938807 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 511
I20260804 14:53:17.938957 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260804 14:53:17.941974 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.130:0
--local_ip_for_outbound_sockets=127.24.6.130
--webserver_interface=127.24.6.130
--webserver_port=0
--tserver_master_addrs=127.24.6.190:46077
--builtin_ntp_servers=127.24.6.148:34955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--consensus_rpc_timeout_ms=15000 with env {}
I20260804 14:53:17.945801 700 heartbeater.cc:347] Connected to a master server at 127.24.6.190:46077
I20260804 14:53:17.946143 700 heartbeater.cc:464] Registering TS with master...
I20260804 14:53:17.946803 700 heartbeater.cc:511] Master 127.24.6.190:46077 requested a full tablet report, sending...
I20260804 14:53:17.948732 427 ts_manager.cc:194] Registered new tserver with Master: bc3e61f1d2fb482f9d6490c298efa2f0 (127.24.6.129:40657)
I20260804 14:53:17.949993 427 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.129:52525
W20260804 14:53:18.073400 706 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260804 14:53:18.073750 706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:18.073820 706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:18.073906 706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:18.078469 706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:18.078670 706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.130
I20260804 14:53:18.084177 706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:34955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=15000
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.24.6.130
--webserver_port=0
--tserver_master_addrs=127.24.6.190:46077
--never_fsync=true
--heap_profile_path=/tmp/kudu.706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.130
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:18.085489 706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:18.087128 706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:18.098300 714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:18.098297 718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:18.098621 706 server_base.cc:1034] running on GCE node
W20260804 14:53:18.098750 716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:18.099375 706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:18.100216 706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:18.101436 706 hybrid_clock.cc:648] HybridClock initialized: now 1785855198101396 us; error 57 us; skew 500 ppm
I20260804 14:53:18.103772 706 webserver.cc:460] Webserver started at http://127.24.6.130:38783/ using document root <none> and password file <none>
I20260804 14:53:18.104486 706 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:18.104594 706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:18.104859 706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:18.109966 706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "da78dd375a274333804236d7d48a51a0"
format_stamp: "Formatted at 2026-08-04 14:53:18 on dist-test-slave-92m1"
I20260804 14:53:18.110878 706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "da78dd375a274333804236d7d48a51a0"
format_stamp: "Formatted at 2026-08-04 14:53:18 on dist-test-slave-92m1"
I20260804 14:53:18.115957 706 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260804 14:53:18.119119 725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:18.120630 706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260804 14:53:18.120811 706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "da78dd375a274333804236d7d48a51a0"
format_stamp: "Formatted at 2026-08-04 14:53:18 on dist-test-slave-92m1"
I20260804 14:53:18.120961 706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:18.143790 706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:18.144622 706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:18.144817 706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:18.145496 706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:53:18.146636 706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:53:18.146693 706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:18.146742 706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:53:18.146759 706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:18.158509 706 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.130:33353
I20260804 14:53:18.158537 891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.130:33353 every 8 connection(s)
I20260804 14:53:18.159705 706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260804 14:53:18.161885 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 706
I20260804 14:53:18.162017 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260804 14:53:18.164762 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.131:0
--local_ip_for_outbound_sockets=127.24.6.131
--webserver_interface=127.24.6.131
--webserver_port=0
--tserver_master_addrs=127.24.6.190:46077
--builtin_ntp_servers=127.24.6.148:34955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--consensus_rpc_timeout_ms=15000 with env {}
I20260804 14:53:18.172058 892 heartbeater.cc:347] Connected to a master server at 127.24.6.190:46077
I20260804 14:53:18.172394 892 heartbeater.cc:464] Registering TS with master...
I20260804 14:53:18.173131 892 heartbeater.cc:511] Master 127.24.6.190:46077 requested a full tablet report, sending...
I20260804 14:53:18.175557 427 ts_manager.cc:194] Registered new tserver with Master: da78dd375a274333804236d7d48a51a0 (127.24.6.130:33353)
I20260804 14:53:18.176438 427 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.130:43437
W20260804 14:53:18.297585 900 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260804 14:53:18.297921 900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:18.297943 900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:18.297978 900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:18.302451 900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:18.302577 900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.131
I20260804 14:53:18.308120 900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:34955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=15000
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.24.6.131
--webserver_port=0
--tserver_master_addrs=127.24.6.190:46077
--never_fsync=true
--heap_profile_path=/tmp/kudu.900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.131
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:18.309329 900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:18.310765 900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:18.321619 907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:18.321632 912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:18.321650 908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:18.322566 900 server_base.cc:1034] running on GCE node
I20260804 14:53:18.323186 900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:18.324048 900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:18.325255 900 hybrid_clock.cc:648] HybridClock initialized: now 1785855198325239 us; error 34 us; skew 500 ppm
I20260804 14:53:18.327749 900 webserver.cc:460] Webserver started at http://127.24.6.131:32883/ using document root <none> and password file <none>
I20260804 14:53:18.328486 900 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:18.328569 900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:18.328840 900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:18.331410 900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b93bddc87eba4178a367d0d49bcc4441"
format_stamp: "Formatted at 2026-08-04 14:53:18 on dist-test-slave-92m1"
I20260804 14:53:18.332193 900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b93bddc87eba4178a367d0d49bcc4441"
format_stamp: "Formatted at 2026-08-04 14:53:18 on dist-test-slave-92m1"
I20260804 14:53:18.337311 900 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260804 14:53:18.340651 919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:18.342175 900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260804 14:53:18.342355 900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b93bddc87eba4178a367d0d49bcc4441"
format_stamp: "Formatted at 2026-08-04 14:53:18 on dist-test-slave-92m1"
I20260804 14:53:18.342505 900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:18.359191 900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:18.360072 900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:18.360316 900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:18.361014 900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:53:18.362138 900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:53:18.362218 900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:18.362296 900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:53:18.362345 900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:18.372906 900 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.131:35407
I20260804 14:53:18.372931 1080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.131:35407 every 8 connection(s)
I20260804 14:53:18.373971 900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260804 14:53:18.374396 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 900
I20260804 14:53:18.374543 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260804 14:53:18.386363 1081 heartbeater.cc:347] Connected to a master server at 127.24.6.190:46077
I20260804 14:53:18.386646 1081 heartbeater.cc:464] Registering TS with master...
I20260804 14:53:18.387336 1081 heartbeater.cc:511] Master 127.24.6.190:46077 requested a full tablet report, sending...
I20260804 14:53:18.388716 427 ts_manager.cc:194] Registered new tserver with Master: b93bddc87eba4178a367d0d49bcc4441 (127.24.6.131:35407)
I20260804 14:53:18.389535 427 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.131:36897
I20260804 14:53:18.389909 24602 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260804 14:53:18.401212 427 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36532:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20260804 14:53:18.404824 427 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260804 14:53:18.424978 604 tablet_service.cc:1467] Processing CreateTablet for tablet 1bdfec46a2f144c8939ead175ee67e0a (DEFAULT_TABLE table=TestTable [id=66112864b0c4475aa5aa91e5bc97b8cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:53:18.426147 604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bdfec46a2f144c8939ead175ee67e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:18.429765 796 tablet_service.cc:1467] Processing CreateTablet for tablet 1bdfec46a2f144c8939ead175ee67e0a (DEFAULT_TABLE table=TestTable [id=66112864b0c4475aa5aa91e5bc97b8cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:53:18.431030 796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bdfec46a2f144c8939ead175ee67e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:18.434057 1110 tablet_bootstrap.cc:492] T 1bdfec46a2f144c8939ead175ee67e0a P bc3e61f1d2fb482f9d6490c298efa2f0: Bootstrap starting.
I20260804 14:53:18.434019 982 tablet_service.cc:1467] Processing CreateTablet for tablet 1bdfec46a2f144c8939ead175ee67e0a (DEFAULT_TABLE table=TestTable [id=66112864b0c4475aa5aa91e5bc97b8cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:53:18.435619 982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bdfec46a2f144c8939ead175ee67e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:18.437292 1110 tablet_bootstrap.cc:654] T 1bdfec46a2f144c8939ead175ee67e0a P bc3e61f1d2fb482f9d6490c298efa2f0: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:18.438815 1110 log.cc:824] T 1bdfec46a2f144c8939ead175ee67e0a P bc3e61f1d2fb482f9d6490c298efa2f0: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:18.438478 1112 tablet_bootstrap.cc:492] T 1bdfec46a2f144c8939ead175ee67e0a P da78dd375a274333804236d7d48a51a0: Bootstrap starting.
I20260804 14:53:18.441888 1112 tablet_bootstrap.cc:654] T 1bdfec46a2f144c8939ead175ee67e0a P da78dd375a274333804236d7d48a51a0: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:18.441964 1110 tablet_bootstrap.cc:492] T 1bdfec46a2f144c8939ead175ee67e0a P bc3e61f1d2fb482f9d6490c298efa2f0: No bootstrap required, opened a new log
I20260804 14:53:18.442230 1110 ts_tablet_manager.cc:1397] T 1bdfec46a2f144c8939ead175ee67e0a P bc3e61f1d2fb482f9d6490c298efa2f0: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260804 14:53:18.442904 1112 log.cc:824] T 1bdfec46a2f144c8939ead175ee67e0a P da78dd375a274333804236d7d48a51a0: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:18.444794 1116 tablet_bootstrap.cc:492] T 1bdfec46a2f144c8939ead175ee67e0a P b93bddc87eba4178a367d0d49bcc4441: Bootstrap starting.
I20260804 14:53:18.446815 1112 tablet_bootstrap.cc:492] T 1bdfec46a2f144c8939ead175ee67e0a P da78dd375a274333804236d7d48a51a0: No bootstrap required, opened a new log
I20260804 14:53:18.447101 1112 ts_tablet_manager.cc:1397] T 1bdfec46a2f144c8939ead175ee67e0a P da78dd375a274333804236d7d48a51a0: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260804 14:53:18.447844 1116 tablet_bootstrap.cc:654] T 1bdfec46a2f144c8939ead175ee67e0a P b93bddc87eba4178a367d0d49bcc4441: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:18.448648 1116 log.cc:824] T 1bdfec46a2f144c8939ead175ee67e0a P b93bddc87eba4178a367d0d49bcc4441: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:18.448725 1110 raft_consensus.cc:348] T 1bdfec46a2f144c8939ead175ee67e0a P bc3e61f1d2fb482f9d6490c298efa2f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b93bddc87eba4178a367d0d49bcc4441" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 35407 } } peers { permanent_uuid: "bc3e61f1d2fb482f9d6490c298efa2f0" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40657 } } peers { permanent_uuid: "da78dd375a274333804236d7d48a51a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33353 } }
I20260804 14:53:18.449144 1110 raft_consensus.cc:729] T 1bdfec46a2f144c8939ead175ee67e0a P bc3e61f1d2fb482f9d6490c298efa2f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc3e61f1d2fb482f9d6490c298efa2f0, State: Initialized, Role: FOLLOWER
I20260804 14:53:18.449565 1110 consensus_queue.cc:260] T 1bdfec46a2f144c8939ead175ee67e0a P bc3e61f1d2fb482f9d6490c298efa2f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b93bddc87eba4178a367d0d49bcc4441" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 35407 } } peers { permanent_uuid: "bc3e61f1d2fb482f9d6490c298efa2f0" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40657 } } peers { permanent_uuid: "da78dd375a274333804236d7d48a51a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33353 } }
I20260804 14:53:18.450299 700 heartbeater.cc:503] Master 127.24.6.190:46077 was elected leader, sending a full tablet report...
I20260804 14:53:18.450788 1110 ts_tablet_manager.cc:1428] T 1bdfec46a2f144c8939ead175ee67e0a P bc3e61f1d2fb482f9d6490c298efa2f0: Time spent starting tablet: real 0.008s user 0.010s sys 0.000s
I20260804 14:53:18.450963 1116 tablet_bootstrap.cc:492] T 1bdfec46a2f144c8939ead175ee67e0a P b93bddc87eba4178a367d0d49bcc4441: No bootstrap required, opened a new log
I20260804 14:53:18.451179 1116 ts_tablet_manager.cc:1397] T 1bdfec46a2f144c8939ead175ee67e0a P b93bddc87eba4178a367d0d49bcc4441: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260804 14:53:18.452311 1112 raft_consensus.cc:348] T 1bdfec46a2f144c8939ead175ee67e0a P da78dd375a274333804236d7d48a51a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b93bddc87eba4178a367d0d49bcc4441" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 35407 } } peers { permanent_uuid: "bc3e61f1d2fb482f9d6490c298efa2f0" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40657 } } peers { permanent_uuid: "da78dd375a274333804236d7d48a51a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33353 } }
I20260804 14:53:18.452688 1112 raft_consensus.cc:729] T 1bdfec46a2f144c8939ead175ee67e0a P da78dd375a274333804236d7d48a51a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da78dd375a274333804236d7d48a51a0, State: Initialized, Role: FOLLOWER
I20260804 14:53:18.453037 1112 consensus_queue.cc:260] T 1bdfec46a2f144c8939ead175ee67e0a P da78dd375a274333804236d7d48a51a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b93bddc87eba4178a367d0d49bcc4441" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 35407 } } peers { permanent_uuid: "bc3e61f1d2fb482f9d6490c298efa2f0" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40657 } } peers { permanent_uuid: "da78dd375a274333804236d7d48a51a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33353 } }
I20260804 14:53:18.453902 892 heartbeater.cc:503] Master 127.24.6.190:46077 was elected leader, sending a full tablet report...
I20260804 14:53:18.454349 1112 ts_tablet_manager.cc:1428] T 1bdfec46a2f144c8939ead175ee67e0a P da78dd375a274333804236d7d48a51a0: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260804 14:53:18.457098 1116 raft_consensus.cc:348] T 1bdfec46a2f144c8939ead175ee67e0a P b93bddc87eba4178a367d0d49bcc4441 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b93bddc87eba4178a367d0d49bcc4441" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 35407 } } peers { permanent_uuid: "bc3e61f1d2fb482f9d6490c298efa2f0" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40657 } } peers { permanent_uuid: "da78dd375a274333804236d7d48a51a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33353 } }
I20260804 14:53:18.457494 1116 raft_consensus.cc:729] T 1bdfec46a2f144c8939ead175ee67e0a P b93bddc87eba4178a367d0d49bcc4441 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b93bddc87eba4178a367d0d49bcc4441, State: Initialized, Role: FOLLOWER
I20260804 14:53:18.457870 1116 consensus_queue.cc:260] T 1bdfec46a2f144c8939ead175ee67e0a P b93bddc87eba4178a367d0d49bcc4441 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b93bddc87eba4178a367d0d49bcc4441" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 35407 } } peers { permanent_uuid: "bc3e61f1d2fb482f9d6490c298efa2f0" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40657 } } peers { permanent_uuid: "da78dd375a274333804236d7d48a51a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33353 } }
I20260804 14:53:18.458700 1081 heartbeater.cc:503] Master 127.24.6.190:46077 was elected leader, sending a full tablet report...
I20260804 14:53:18.459172 1116 ts_tablet_manager.cc:1428] T 1bdfec46a2f144c8939ead175ee67e0a P b93bddc87eba4178a367d0d49bcc4441: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260804 14:53:18.471647 24602 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260804 14:53:18.473618 24602 ts_itest-base.cc:233] Waiting for 1 tablets on tserver bc3e61f1d2fb482f9d6490c298efa2f0 to finish bootstrapping
I20260804 14:53:18.481076 24602 ts_itest-base.cc:233] Waiting for 1 tablets on tserver da78dd375a274333804236d7d48a51a0 to finish bootstrapping
I20260804 14:53:18.488329 24602 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b93bddc87eba4178a367d0d49bcc4441 to finish bootstrapping
I20260804 14:53:18.501096 827 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1bdfec46a2f144c8939ead175ee67e0a"
dest_uuid: "da78dd375a274333804236d7d48a51a0"
from {username='slave'} at 127.0.0.1:37238
I20260804 14:53:18.501565 827 raft_consensus.cc:482] T 1bdfec46a2f144c8939ead175ee67e0a P da78dd375a274333804236d7d48a51a0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260804 14:53:18.501726 827 raft_consensus.cc:3037] T 1bdfec46a2f144c8939ead175ee67e0a P da78dd375a274333804236d7d48a51a0 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:18.502815 827 raft_consensus.cc:504] T 1bdfec46a2f144c8939ead175ee67e0a P da78dd375a274333804236d7d48a51a0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b93bddc87eba4178a367d0d49bcc4441" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 35407 } } peers { permanent_uuid: "bc3e61f1d2fb482f9d6490c298efa2f0" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40657 } } peers { permanent_uuid: "da78dd375a274333804236d7d48a51a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33353 } }
I20260804 14:53:18.504204 827 leader_election.cc:288] T 1bdfec46a2f144c8939ead175ee67e0a P da78dd375a274333804236d7d48a51a0 [CANDIDATE]: Term 1 election: Requested vote from peers b93bddc87eba4178a367d0d49bcc4441 (127.24.6.131:35407), bc3e61f1d2fb482f9d6490c298efa2f0 (127.24.6.129:40657)
I20260804 14:53:18.509752 632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bdfec46a2f144c8939ead175ee67e0a" candidate_uuid: "da78dd375a274333804236d7d48a51a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bc3e61f1d2fb482f9d6490c298efa2f0"
I20260804 14:53:18.509967 1011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bdfec46a2f144c8939ead175ee67e0a" candidate_uuid: "da78dd375a274333804236d7d48a51a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b93bddc87eba4178a367d0d49bcc4441"
I20260804 14:53:18.510169 632 raft_consensus.cc:3037] T 1bdfec46a2f144c8939ead175ee67e0a P bc3e61f1d2fb482f9d6490c298efa2f0 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:18.510272 1011 raft_consensus.cc:3037] T 1bdfec46a2f144c8939ead175ee67e0a P b93bddc87eba4178a367d0d49bcc4441 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:18.511822 1011 raft_consensus.cc:2454] T 1bdfec46a2f144c8939ead175ee67e0a P b93bddc87eba4178a367d0d49bcc4441 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da78dd375a274333804236d7d48a51a0 in term 1.
I20260804 14:53:18.511827 632 raft_consensus.cc:2454] T 1bdfec46a2f144c8939ead175ee67e0a P bc3e61f1d2fb482f9d6490c298efa2f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da78dd375a274333804236d7d48a51a0 in term 1.
I20260804 14:53:18.512418 726 leader_election.cc:302] T 1bdfec46a2f144c8939ead175ee67e0a P da78dd375a274333804236d7d48a51a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b93bddc87eba4178a367d0d49bcc4441, da78dd375a274333804236d7d48a51a0; no voters:
I20260804 14:53:18.512810 1120 raft_consensus.cc:2781] T 1bdfec46a2f144c8939ead175ee67e0a P da78dd375a274333804236d7d48a51a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:18.513085 1120 raft_consensus.cc:686] T 1bdfec46a2f144c8939ead175ee67e0a P da78dd375a274333804236d7d48a51a0 [term 1 LEADER]: Becoming Leader. State: Replica: da78dd375a274333804236d7d48a51a0, State: Running, Role: LEADER
I20260804 14:53:18.513602 1120 consensus_queue.cc:237] T 1bdfec46a2f144c8939ead175ee67e0a P da78dd375a274333804236d7d48a51a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b93bddc87eba4178a367d0d49bcc4441" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 35407 } } peers { permanent_uuid: "bc3e61f1d2fb482f9d6490c298efa2f0" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40657 } } peers { permanent_uuid: "da78dd375a274333804236d7d48a51a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33353 } }
I20260804 14:53:18.517627 427 catalog_manager.cc:5495] T 1bdfec46a2f144c8939ead175ee67e0a P da78dd375a274333804236d7d48a51a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to da78dd375a274333804236d7d48a51a0 (127.24.6.130). New cstate: current_term: 1 leader_uuid: "da78dd375a274333804236d7d48a51a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b93bddc87eba4178a367d0d49bcc4441" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 35407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc3e61f1d2fb482f9d6490c298efa2f0" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da78dd375a274333804236d7d48a51a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33353 } health_report { overall_health: HEALTHY } } }
W20260804 14:53:18.661170 895 tablet.cc:2369] T 1bdfec46a2f144c8939ead175ee67e0a P da78dd375a274333804236d7d48a51a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260804 14:53:19.540138 827 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1bdfec46a2f144c8939ead175ee67e0a"
dest_uuid: "da78dd375a274333804236d7d48a51a0"
from {username='slave'} at 127.0.0.1:37238
I20260804 14:53:19.540530 827 raft_consensus.cc:580] T 1bdfec46a2f144c8939ead175ee67e0a P da78dd375a274333804236d7d48a51a0 [term 1 LEADER]: Received request to step down
I20260804 14:53:19.540614 827 raft_consensus.cc:3032] T 1bdfec46a2f144c8939ead175ee67e0a P da78dd375a274333804236d7d48a51a0 [term 1 LEADER]: Stepping down as leader of term 1
I20260804 14:53:19.540688 827 raft_consensus.cc:729] T 1bdfec46a2f144c8939ead175ee67e0a P da78dd375a274333804236d7d48a51a0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: da78dd375a274333804236d7d48a51a0, State: Running, Role: LEADER
I20260804 14:53:19.540910 827 consensus_queue.cc:260] T 1bdfec46a2f144c8939ead175ee67e0a P da78dd375a274333804236d7d48a51a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b93bddc87eba4178a367d0d49bcc4441" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 35407 } } peers { permanent_uuid: "bc3e61f1d2fb482f9d6490c298efa2f0" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40657 } } peers { permanent_uuid: "da78dd375a274333804236d7d48a51a0" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 33353 } }
I20260804 14:53:19.541138 827 raft_consensus.cc:3037] T 1bdfec46a2f144c8939ead175ee67e0a P da78dd375a274333804236d7d48a51a0 [term 1 FOLLOWER]: Advancing to term 2
I20260804 14:53:19.542413 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 511
W20260804 14:53:19.555439 729 connection.cc:535] client connection to 127.24.6.129:40657 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260804 14:53:19.555838 729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260804 14:53:19.555866 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 706
I20260804 14:53:19.572103 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 900
I20260804 14:53:19.586014 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 384
2026-08-04T14:53:19Z chronyd exiting
[ OK ] RaftConsensusElectionITest.StepDownWithSlowFollower (2207 ms)
[ RUN ] RaftConsensusElectionITest.ElectPendingVoter
I20260804 14:53:19.618999 24602 test_util.cc:268] Using random seed: -791992403
I20260804 14:53:19.620013 24602 ts_itest-base.cc:112] Starting cluster with:
I20260804 14:53:19.620059 24602 ts_itest-base.cc:113] --------------
I20260804 14:53:19.620071 24602 ts_itest-base.cc:114] 5 tablet servers
I20260804 14:53:19.620082 24602 ts_itest-base.cc:115] 5 replicas per TS
I20260804 14:53:19.620093 24602 ts_itest-base.cc:116] --------------
2026-08-04T14:53:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-04T14:53:19Z Disabled control of system clock
I20260804 14:53:19.632476 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.6.190:36209
--webserver_interface=127.24.6.190
--webserver_port=0
--builtin_ntp_servers=127.24.6.148:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.6.190:36209
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20260804 14:53:19.755237 1145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:19.755625 1145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:19.755702 1145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:19.760322 1145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260804 14:53:19.760433 1145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:19.760465 1145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260804 14:53:19.760509 1145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260804 14:53:19.765614 1145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.24.6.190:36209
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.6.190:36209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.24.6.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.1145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:19.766979 1145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:19.768584 1145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:19.778122 1154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:19.778175 1151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:19.778165 1152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:19.778468 1145 server_base.cc:1034] running on GCE node
I20260804 14:53:19.779093 1145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:19.780397 1145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:19.781589 1145 hybrid_clock.cc:648] HybridClock initialized: now 1785855199781574 us; error 27 us; skew 500 ppm
I20260804 14:53:19.791874 1145 webserver.cc:460] Webserver started at http://127.24.6.190:45295/ using document root <none> and password file <none>
I20260804 14:53:19.792762 1145 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:19.792905 1145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:19.793258 1145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:19.795976 1145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d0916e880090404aa6a0bf4ca970ac79"
format_stamp: "Formatted at 2026-08-04 14:53:19 on dist-test-slave-92m1"
I20260804 14:53:19.796756 1145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d0916e880090404aa6a0bf4ca970ac79"
format_stamp: "Formatted at 2026-08-04 14:53:19 on dist-test-slave-92m1"
I20260804 14:53:19.801976 1145 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260804 14:53:19.805277 1161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:19.806762 1145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260804 14:53:19.806967 1145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d0916e880090404aa6a0bf4ca970ac79"
format_stamp: "Formatted at 2026-08-04 14:53:19 on dist-test-slave-92m1"
I20260804 14:53:19.807441 1145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:19.822871 1145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:19.823834 1145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:19.824133 1145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:19.832582 1145 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.190:36209
I20260804 14:53:19.832609 1238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.190:36209 every 8 connection(s)
I20260804 14:53:19.833695 1145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260804 14:53:19.837881 1239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:19.840734 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 1145
I20260804 14:53:19.840859 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260804 14:53:19.845538 1239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0916e880090404aa6a0bf4ca970ac79: Bootstrap starting.
I20260804 14:53:19.848963 1239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0916e880090404aa6a0bf4ca970ac79: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:19.850108 1239 log.cc:824] T 00000000000000000000000000000000 P d0916e880090404aa6a0bf4ca970ac79: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:19.853211 1239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0916e880090404aa6a0bf4ca970ac79: No bootstrap required, opened a new log
I20260804 14:53:19.858278 1239 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0916e880090404aa6a0bf4ca970ac79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0916e880090404aa6a0bf4ca970ac79" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 36209 } }
I20260804 14:53:19.858621 1239 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0916e880090404aa6a0bf4ca970ac79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:53:19.858665 1239 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0916e880090404aa6a0bf4ca970ac79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0916e880090404aa6a0bf4ca970ac79, State: Initialized, Role: FOLLOWER
I20260804 14:53:19.859133 1239 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0916e880090404aa6a0bf4ca970ac79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0916e880090404aa6a0bf4ca970ac79" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 36209 } }
I20260804 14:53:19.859359 1239 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0916e880090404aa6a0bf4ca970ac79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260804 14:53:19.859460 1239 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0916e880090404aa6a0bf4ca970ac79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260804 14:53:19.859578 1239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0916e880090404aa6a0bf4ca970ac79 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:19.860711 1239 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0916e880090404aa6a0bf4ca970ac79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0916e880090404aa6a0bf4ca970ac79" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 36209 } }
I20260804 14:53:19.861177 1239 leader_election.cc:302] T 00000000000000000000000000000000 P d0916e880090404aa6a0bf4ca970ac79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0916e880090404aa6a0bf4ca970ac79; no voters:
I20260804 14:53:19.861732 1239 leader_election.cc:288] T 00000000000000000000000000000000 P d0916e880090404aa6a0bf4ca970ac79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260804 14:53:19.861855 1247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0916e880090404aa6a0bf4ca970ac79 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:19.862124 1247 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0916e880090404aa6a0bf4ca970ac79 [term 1 LEADER]: Becoming Leader. State: Replica: d0916e880090404aa6a0bf4ca970ac79, State: Running, Role: LEADER
I20260804 14:53:19.862495 1247 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0916e880090404aa6a0bf4ca970ac79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0916e880090404aa6a0bf4ca970ac79" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 36209 } }
I20260804 14:53:19.863417 1239 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0916e880090404aa6a0bf4ca970ac79 [sys.catalog]: configured and running, proceeding with master startup.
I20260804 14:53:19.864621 1250 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0916e880090404aa6a0bf4ca970ac79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0916e880090404aa6a0bf4ca970ac79. Latest consensus state: current_term: 1 leader_uuid: "d0916e880090404aa6a0bf4ca970ac79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0916e880090404aa6a0bf4ca970ac79" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 36209 } } }
I20260804 14:53:19.864785 1250 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0916e880090404aa6a0bf4ca970ac79 [sys.catalog]: This master's current role is: LEADER
I20260804 14:53:19.865123 1248 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0916e880090404aa6a0bf4ca970ac79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0916e880090404aa6a0bf4ca970ac79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0916e880090404aa6a0bf4ca970ac79" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 36209 } } }
I20260804 14:53:19.865299 1248 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0916e880090404aa6a0bf4ca970ac79 [sys.catalog]: This master's current role is: LEADER
I20260804 14:53:19.866370 1261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260804 14:53:19.869762 1261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260804 14:53:19.876433 1261 catalog_manager.cc:1347] Generated new cluster ID: d0dcbaf8d0e1488b8e5b68de837e65f1
I20260804 14:53:19.876559 1261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260804 14:53:19.899379 1261 catalog_manager.cc:1370] Generated new certificate authority record
I20260804 14:53:19.900837 1261 catalog_manager.cc:1504] Loading token signing keys...
I20260804 14:53:19.909204 1261 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0916e880090404aa6a0bf4ca970ac79: Generated new TSK 0
I20260804 14:53:19.910317 1261 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260804 14:53:19.924103 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.129:0
--local_ip_for_outbound_sockets=127.24.6.129
--webserver_interface=127.24.6.129
--webserver_port=0
--tserver_master_addrs=127.24.6.190:36209
--builtin_ntp_servers=127.24.6.148:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20260804 14:53:20.057686 1272 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260804 14:53:20.058070 1272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:20.058095 1272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:20.058132 1272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:20.063674 1272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:20.063818 1272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.129
I20260804 14:53:20.070089 1272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.24.6.129
--webserver_port=0
--tserver_master_addrs=127.24.6.190:36209
--never_fsync=true
--heap_profile_path=/tmp/kudu.1272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.129
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:20.071583 1272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:20.073414 1272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:20.086407 1280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:20.086388 1277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:20.086589 1272 server_base.cc:1034] running on GCE node
W20260804 14:53:20.086388 1278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:20.087421 1272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:20.088269 1272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:20.089488 1272 hybrid_clock.cc:648] HybridClock initialized: now 1785855200089469 us; error 33 us; skew 500 ppm
I20260804 14:53:20.092048 1272 webserver.cc:460] Webserver started at http://127.24.6.129:34539/ using document root <none> and password file <none>
I20260804 14:53:20.092905 1272 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:20.093029 1272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:20.093341 1272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:20.095731 1272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "6828849eb09e4f7d8434388177605134"
format_stamp: "Formatted at 2026-08-04 14:53:20 on dist-test-slave-92m1"
I20260804 14:53:20.096484 1272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "6828849eb09e4f7d8434388177605134"
format_stamp: "Formatted at 2026-08-04 14:53:20 on dist-test-slave-92m1"
I20260804 14:53:20.101094 1272 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260804 14:53:20.104489 1288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:20.105978 1272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260804 14:53:20.106207 1272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6828849eb09e4f7d8434388177605134"
format_stamp: "Formatted at 2026-08-04 14:53:20 on dist-test-slave-92m1"
I20260804 14:53:20.106346 1272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:20.136485 1272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:20.137341 1272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:20.137574 1272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:20.138281 1272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:53:20.139451 1272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:53:20.139508 1272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:20.139585 1272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:53:20.139617 1272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:20.150511 1272 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.129:34733
I20260804 14:53:20.150584 1443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.129:34733 every 8 connection(s)
I20260804 14:53:20.151664 1272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260804 14:53:20.154074 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 1272
I20260804 14:53:20.154229 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260804 14:53:20.157111 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.130:0
--local_ip_for_outbound_sockets=127.24.6.130
--webserver_interface=127.24.6.130
--webserver_port=0
--tserver_master_addrs=127.24.6.190:36209
--builtin_ntp_servers=127.24.6.148:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260804 14:53:20.166230 1444 heartbeater.cc:347] Connected to a master server at 127.24.6.190:36209
I20260804 14:53:20.166571 1444 heartbeater.cc:464] Registering TS with master...
I20260804 14:53:20.167407 1444 heartbeater.cc:511] Master 127.24.6.190:36209 requested a full tablet report, sending...
I20260804 14:53:20.169229 1182 ts_manager.cc:194] Registered new tserver with Master: 6828849eb09e4f7d8434388177605134 (127.24.6.129:34733)
I20260804 14:53:20.170468 1182 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.129:46875
W20260804 14:53:20.294335 1451 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260804 14:53:20.294708 1451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:20.294732 1451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:20.294827 1451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:20.299831 1451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:20.299973 1451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.130
I20260804 14:53:20.306064 1451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.24.6.130
--webserver_port=0
--tserver_master_addrs=127.24.6.190:36209
--never_fsync=true
--heap_profile_path=/tmp/kudu.1451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.130
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:20.307483 1451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:20.309264 1451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:20.321125 1463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:20.321291 1458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:20.321305 1459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:20.321440 1451 server_base.cc:1034] running on GCE node
I20260804 14:53:20.322217 1451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:20.322984 1451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:20.324186 1451 hybrid_clock.cc:648] HybridClock initialized: now 1785855200324157 us; error 51 us; skew 500 ppm
I20260804 14:53:20.327018 1451 webserver.cc:460] Webserver started at http://127.24.6.130:37289/ using document root <none> and password file <none>
I20260804 14:53:20.327875 1451 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:20.327962 1451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:20.328579 1451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:20.332834 1451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "7bea20298f68472fbbb0a631dc011bc3"
format_stamp: "Formatted at 2026-08-04 14:53:20 on dist-test-slave-92m1"
I20260804 14:53:20.334113 1451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "7bea20298f68472fbbb0a631dc011bc3"
format_stamp: "Formatted at 2026-08-04 14:53:20 on dist-test-slave-92m1"
I20260804 14:53:20.343466 1451 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.010s sys 0.000s
I20260804 14:53:20.346868 1473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:20.349406 1451 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260804 14:53:20.349625 1451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "7bea20298f68472fbbb0a631dc011bc3"
format_stamp: "Formatted at 2026-08-04 14:53:20 on dist-test-slave-92m1"
I20260804 14:53:20.349797 1451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:20.368093 1451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:20.368919 1451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:20.369138 1451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:20.369861 1451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:53:20.371006 1451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:53:20.371057 1451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:20.371104 1451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:53:20.371160 1451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:20.382270 1451 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.130:39539
I20260804 14:53:20.382288 1646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.130:39539 every 8 connection(s)
I20260804 14:53:20.383457 1451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260804 14:53:20.387982 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 1451
I20260804 14:53:20.388121 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260804 14:53:20.390676 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.131:0
--local_ip_for_outbound_sockets=127.24.6.131
--webserver_interface=127.24.6.131
--webserver_port=0
--tserver_master_addrs=127.24.6.190:36209
--builtin_ntp_servers=127.24.6.148:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260804 14:53:20.397333 1647 heartbeater.cc:347] Connected to a master server at 127.24.6.190:36209
I20260804 14:53:20.397651 1647 heartbeater.cc:464] Registering TS with master...
I20260804 14:53:20.398288 1647 heartbeater.cc:511] Master 127.24.6.190:36209 requested a full tablet report, sending...
I20260804 14:53:20.399700 1182 ts_manager.cc:194] Registered new tserver with Master: 7bea20298f68472fbbb0a631dc011bc3 (127.24.6.130:39539)
I20260804 14:53:20.400633 1182 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.130:44803
W20260804 14:53:20.517514 1654 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260804 14:53:20.517864 1654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:20.517887 1654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:20.517922 1654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:20.523303 1654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:20.523447 1654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.131
I20260804 14:53:20.529155 1654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.24.6.131
--webserver_port=0
--tserver_master_addrs=127.24.6.190:36209
--never_fsync=true
--heap_profile_path=/tmp/kudu.1654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.131
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:20.530400 1654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:20.531936 1654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:20.542771 1662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:20.542773 1661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:20.542773 1654 server_base.cc:1034] running on GCE node
W20260804 14:53:20.542791 1664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:20.543656 1654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:20.544310 1654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:20.545492 1654 hybrid_clock.cc:648] HybridClock initialized: now 1785855200545474 us; error 27 us; skew 500 ppm
I20260804 14:53:20.547693 1654 webserver.cc:460] Webserver started at http://127.24.6.131:34871/ using document root <none> and password file <none>
I20260804 14:53:20.548358 1654 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:20.548436 1654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:20.548714 1654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:20.550774 1654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e2ba1c7294da4cb89d31ce0113c187f4"
format_stamp: "Formatted at 2026-08-04 14:53:20 on dist-test-slave-92m1"
I20260804 14:53:20.551451 1654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e2ba1c7294da4cb89d31ce0113c187f4"
format_stamp: "Formatted at 2026-08-04 14:53:20 on dist-test-slave-92m1"
I20260804 14:53:20.555702 1654 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260804 14:53:20.558537 1675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:20.560083 1654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260804 14:53:20.560256 1654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e2ba1c7294da4cb89d31ce0113c187f4"
format_stamp: "Formatted at 2026-08-04 14:53:20 on dist-test-slave-92m1"
I20260804 14:53:20.560447 1654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:20.583401 1654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:20.584254 1654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:20.584486 1654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:20.585153 1654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:53:20.586202 1654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:53:20.586277 1654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:20.586370 1654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:53:20.586421 1654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:20.596623 1654 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.131:42643
I20260804 14:53:20.596649 1842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.131:42643 every 8 connection(s)
I20260804 14:53:20.597844 1654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260804 14:53:20.600431 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 1654
I20260804 14:53:20.600586 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260804 14:53:20.603503 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.132:0
--local_ip_for_outbound_sockets=127.24.6.132
--webserver_interface=127.24.6.132
--webserver_port=0
--tserver_master_addrs=127.24.6.190:36209
--builtin_ntp_servers=127.24.6.148:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260804 14:53:20.611434 1843 heartbeater.cc:347] Connected to a master server at 127.24.6.190:36209
I20260804 14:53:20.611783 1843 heartbeater.cc:464] Registering TS with master...
I20260804 14:53:20.612676 1843 heartbeater.cc:511] Master 127.24.6.190:36209 requested a full tablet report, sending...
I20260804 14:53:20.614235 1182 ts_manager.cc:194] Registered new tserver with Master: e2ba1c7294da4cb89d31ce0113c187f4 (127.24.6.131:42643)
I20260804 14:53:20.614889 1182 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.131:56681
W20260804 14:53:20.730711 1847 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260804 14:53:20.731046 1847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:20.731074 1847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:20.731109 1847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:20.735409 1847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:20.735519 1847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.132
I20260804 14:53:20.740688 1847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.24.6.132
--webserver_port=0
--tserver_master_addrs=127.24.6.190:36209
--never_fsync=true
--heap_profile_path=/tmp/kudu.1847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.132
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:20.741920 1847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:20.743477 1847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:20.754104 1855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:20.754220 1847 server_base.cc:1034] running on GCE node
W20260804 14:53:20.754238 1852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:20.754314 1853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:20.755007 1847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:20.755823 1847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:20.757020 1847 hybrid_clock.cc:648] HybridClock initialized: now 1785855200756983 us; error 58 us; skew 500 ppm
I20260804 14:53:20.759373 1847 webserver.cc:460] Webserver started at http://127.24.6.132:42595/ using document root <none> and password file <none>
I20260804 14:53:20.760044 1847 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:20.760155 1847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:20.760416 1847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:20.762562 1847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "3b87eee3311049e3873f76a8b7abd9ee"
format_stamp: "Formatted at 2026-08-04 14:53:20 on dist-test-slave-92m1"
I20260804 14:53:20.763301 1847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "3b87eee3311049e3873f76a8b7abd9ee"
format_stamp: "Formatted at 2026-08-04 14:53:20 on dist-test-slave-92m1"
I20260804 14:53:20.767997 1847 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260804 14:53:20.771111 1862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:20.772677 1847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260804 14:53:20.772840 1847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "3b87eee3311049e3873f76a8b7abd9ee"
format_stamp: "Formatted at 2026-08-04 14:53:20 on dist-test-slave-92m1"
I20260804 14:53:20.772974 1847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:20.788246 1847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:20.789024 1847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:20.789242 1847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:20.789881 1847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:53:20.790905 1847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:53:20.790979 1847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:20.791056 1847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:53:20.791093 1847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:20.800840 1847 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.132:34633
I20260804 14:53:20.800870 2022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.132:34633 every 8 connection(s)
I20260804 14:53:20.801910 1847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20260804 14:53:20.802735 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 1847
I20260804 14:53:20.802875 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20260804 14:53:20.805608 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.133:0
--local_ip_for_outbound_sockets=127.24.6.133
--webserver_interface=127.24.6.133
--webserver_port=0
--tserver_master_addrs=127.24.6.190:36209
--builtin_ntp_servers=127.24.6.148:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260804 14:53:20.815253 2023 heartbeater.cc:347] Connected to a master server at 127.24.6.190:36209
I20260804 14:53:20.815591 2023 heartbeater.cc:464] Registering TS with master...
I20260804 14:53:20.816272 2023 heartbeater.cc:511] Master 127.24.6.190:36209 requested a full tablet report, sending...
I20260804 14:53:20.817584 1182 ts_manager.cc:194] Registered new tserver with Master: 3b87eee3311049e3873f76a8b7abd9ee (127.24.6.132:34633)
I20260804 14:53:20.818502 1182 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.132:58489
W20260804 14:53:20.932125 2028 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260804 14:53:20.932437 2028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:20.932458 2028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:20.932492 2028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:20.936844 2028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:20.936969 2028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.133
I20260804 14:53:20.942103 2028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.24.6.133
--webserver_port=0
--tserver_master_addrs=127.24.6.190:36209
--never_fsync=true
--heap_profile_path=/tmp/kudu.2028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.133
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:20.943403 2028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:20.944876 2028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:20.955497 2037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:20.955528 2035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:20.955518 2040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:20.955753 2028 server_base.cc:1034] running on GCE node
I20260804 14:53:20.956346 2028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:20.957013 2028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:20.958186 2028 hybrid_clock.cc:648] HybridClock initialized: now 1785855200958169 us; error 35 us; skew 500 ppm
I20260804 14:53:20.960340 2028 webserver.cc:460] Webserver started at http://127.24.6.133:41235/ using document root <none> and password file <none>
I20260804 14:53:20.961005 2028 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:20.961076 2028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:20.961315 2028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:20.963470 2028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "32500bcfa932410ab26501b83151acb3"
format_stamp: "Formatted at 2026-08-04 14:53:20 on dist-test-slave-92m1"
I20260804 14:53:20.964236 2028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "32500bcfa932410ab26501b83151acb3"
format_stamp: "Formatted at 2026-08-04 14:53:20 on dist-test-slave-92m1"
I20260804 14:53:20.968715 2028 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260804 14:53:20.971910 2047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:20.973485 2028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260804 14:53:20.973670 2028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "32500bcfa932410ab26501b83151acb3"
format_stamp: "Formatted at 2026-08-04 14:53:20 on dist-test-slave-92m1"
I20260804 14:53:20.973834 2028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:20.996125 2028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:20.996908 2028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:20.997094 2028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:20.997705 2028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:53:20.998852 2028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:53:20.998907 2028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:20.998955 2028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:53:20.998974 2028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:21.009542 2028 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.133:35213
I20260804 14:53:21.009562 2215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.133:35213 every 8 connection(s)
I20260804 14:53:21.010700 2028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20260804 14:53:21.014768 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 2028
I20260804 14:53:21.014940 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20260804 14:53:21.022594 2216 heartbeater.cc:347] Connected to a master server at 127.24.6.190:36209
I20260804 14:53:21.022853 2216 heartbeater.cc:464] Registering TS with master...
I20260804 14:53:21.023444 2216 heartbeater.cc:511] Master 127.24.6.190:36209 requested a full tablet report, sending...
I20260804 14:53:21.024567 1182 ts_manager.cc:194] Registered new tserver with Master: 32500bcfa932410ab26501b83151acb3 (127.24.6.133:35213)
I20260804 14:53:21.025169 1182 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.133:43145
I20260804 14:53:21.029714 24602 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20260804 14:53:21.040401 1182 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46404:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20260804 14:53:21.043709 1182 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 6 tablet servers would be needed, 5 are available. Consider bringing up more tablet servers.
I20260804 14:53:21.064033 1356 tablet_service.cc:1467] Processing CreateTablet for tablet 5618b8f4bcf3448991f469edffb80405 (DEFAULT_TABLE table=TestTable [id=03d508cb5c8c493ea5236e06d7a27da2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:53:21.064265 2115 tablet_service.cc:1467] Processing CreateTablet for tablet 5618b8f4bcf3448991f469edffb80405 (DEFAULT_TABLE table=TestTable [id=03d508cb5c8c493ea5236e06d7a27da2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:53:21.064378 1746 tablet_service.cc:1467] Processing CreateTablet for tablet 5618b8f4bcf3448991f469edffb80405 (DEFAULT_TABLE table=TestTable [id=03d508cb5c8c493ea5236e06d7a27da2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:53:21.065429 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5618b8f4bcf3448991f469edffb80405. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:21.065429 1746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5618b8f4bcf3448991f469edffb80405. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:21.065429 2115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5618b8f4bcf3448991f469edffb80405. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:21.065519 1550 tablet_service.cc:1467] Processing CreateTablet for tablet 5618b8f4bcf3448991f469edffb80405 (DEFAULT_TABLE table=TestTable [id=03d508cb5c8c493ea5236e06d7a27da2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:53:21.066489 1550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5618b8f4bcf3448991f469edffb80405. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:21.074282 2253 tablet_bootstrap.cc:492] T 5618b8f4bcf3448991f469edffb80405 P 32500bcfa932410ab26501b83151acb3: Bootstrap starting.
I20260804 14:53:21.074282 2252 tablet_bootstrap.cc:492] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3: Bootstrap starting.
I20260804 14:53:21.076933 2252 tablet_bootstrap.cc:654] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:21.076946 2253 tablet_bootstrap.cc:654] T 5618b8f4bcf3448991f469edffb80405 P 32500bcfa932410ab26501b83151acb3: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:21.077956 2252 log.cc:824] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:21.078359 2253 log.cc:824] T 5618b8f4bcf3448991f469edffb80405 P 32500bcfa932410ab26501b83151acb3: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:21.078919 2250 tablet_bootstrap.cc:492] T 5618b8f4bcf3448991f469edffb80405 P e2ba1c7294da4cb89d31ce0113c187f4: Bootstrap starting.
I20260804 14:53:21.081200 2253 tablet_bootstrap.cc:492] T 5618b8f4bcf3448991f469edffb80405 P 32500bcfa932410ab26501b83151acb3: No bootstrap required, opened a new log
I20260804 14:53:21.081436 2253 ts_tablet_manager.cc:1397] T 5618b8f4bcf3448991f469edffb80405 P 32500bcfa932410ab26501b83151acb3: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260804 14:53:21.081632 2250 tablet_bootstrap.cc:654] T 5618b8f4bcf3448991f469edffb80405 P e2ba1c7294da4cb89d31ce0113c187f4: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:21.082509 2250 log.cc:824] T 5618b8f4bcf3448991f469edffb80405 P e2ba1c7294da4cb89d31ce0113c187f4: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:21.083390 2251 tablet_bootstrap.cc:492] T 5618b8f4bcf3448991f469edffb80405 P 6828849eb09e4f7d8434388177605134: Bootstrap starting.
I20260804 14:53:21.084542 2252 tablet_bootstrap.cc:492] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3: No bootstrap required, opened a new log
I20260804 14:53:21.084795 2252 ts_tablet_manager.cc:1397] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s
I20260804 14:53:21.085834 2251 tablet_bootstrap.cc:654] T 5618b8f4bcf3448991f469edffb80405 P 6828849eb09e4f7d8434388177605134: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:21.085798 1935 tablet_service.cc:1467] Processing CreateTablet for tablet 5618b8f4bcf3448991f469edffb80405 (DEFAULT_TABLE table=TestTable [id=03d508cb5c8c493ea5236e06d7a27da2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:53:21.086694 2251 log.cc:824] T 5618b8f4bcf3448991f469edffb80405 P 6828849eb09e4f7d8434388177605134: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:21.087070 1935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5618b8f4bcf3448991f469edffb80405. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:21.089033 2251 tablet_bootstrap.cc:492] T 5618b8f4bcf3448991f469edffb80405 P 6828849eb09e4f7d8434388177605134: No bootstrap required, opened a new log
I20260804 14:53:21.089131 2252 raft_consensus.cc:348] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } } peers { permanent_uuid: "32500bcfa932410ab26501b83151acb3" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 35213 } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } }
I20260804 14:53:21.089450 2252 raft_consensus.cc:729] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bea20298f68472fbbb0a631dc011bc3, State: Initialized, Role: FOLLOWER
I20260804 14:53:21.089807 2252 consensus_queue.cc:260] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } } peers { permanent_uuid: "32500bcfa932410ab26501b83151acb3" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 35213 } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } }
I20260804 14:53:21.090473 2253 raft_consensus.cc:348] T 5618b8f4bcf3448991f469edffb80405 P 32500bcfa932410ab26501b83151acb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } } peers { permanent_uuid: "32500bcfa932410ab26501b83151acb3" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 35213 } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } }
I20260804 14:53:21.090786 2253 raft_consensus.cc:729] T 5618b8f4bcf3448991f469edffb80405 P 32500bcfa932410ab26501b83151acb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32500bcfa932410ab26501b83151acb3, State: Initialized, Role: FOLLOWER
I20260804 14:53:21.090837 2251 ts_tablet_manager.cc:1397] T 5618b8f4bcf3448991f469edffb80405 P 6828849eb09e4f7d8434388177605134: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260804 14:53:21.090973 2252 ts_tablet_manager.cc:1428] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260804 14:53:21.091394 1647 heartbeater.cc:503] Master 127.24.6.190:36209 was elected leader, sending a full tablet report...
I20260804 14:53:21.093851 2250 tablet_bootstrap.cc:492] T 5618b8f4bcf3448991f469edffb80405 P e2ba1c7294da4cb89d31ce0113c187f4: No bootstrap required, opened a new log
I20260804 14:53:21.094087 2250 ts_tablet_manager.cc:1397] T 5618b8f4bcf3448991f469edffb80405 P e2ba1c7294da4cb89d31ce0113c187f4: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20260804 14:53:21.095606 2260 tablet_bootstrap.cc:492] T 5618b8f4bcf3448991f469edffb80405 P 3b87eee3311049e3873f76a8b7abd9ee: Bootstrap starting.
I20260804 14:53:21.096078 2251 raft_consensus.cc:348] T 5618b8f4bcf3448991f469edffb80405 P 6828849eb09e4f7d8434388177605134 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } } peers { permanent_uuid: "32500bcfa932410ab26501b83151acb3" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 35213 } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } }
I20260804 14:53:21.096393 2251 raft_consensus.cc:729] T 5618b8f4bcf3448991f469edffb80405 P 6828849eb09e4f7d8434388177605134 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6828849eb09e4f7d8434388177605134, State: Initialized, Role: FOLLOWER
I20260804 14:53:21.096735 2251 consensus_queue.cc:260] T 5618b8f4bcf3448991f469edffb80405 P 6828849eb09e4f7d8434388177605134 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } } peers { permanent_uuid: "32500bcfa932410ab26501b83151acb3" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 35213 } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } }
I20260804 14:53:21.097860 2251 ts_tablet_manager.cc:1428] T 5618b8f4bcf3448991f469edffb80405 P 6828849eb09e4f7d8434388177605134: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260804 14:53:21.098297 1444 heartbeater.cc:503] Master 127.24.6.190:36209 was elected leader, sending a full tablet report...
I20260804 14:53:21.098479 2260 tablet_bootstrap.cc:654] T 5618b8f4bcf3448991f469edffb80405 P 3b87eee3311049e3873f76a8b7abd9ee: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:21.099519 2260 log.cc:824] T 5618b8f4bcf3448991f469edffb80405 P 3b87eee3311049e3873f76a8b7abd9ee: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:21.091127 2253 consensus_queue.cc:260] T 5618b8f4bcf3448991f469edffb80405 P 32500bcfa932410ab26501b83151acb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } } peers { permanent_uuid: "32500bcfa932410ab26501b83151acb3" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 35213 } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } }
I20260804 14:53:21.101217 2253 ts_tablet_manager.cc:1428] T 5618b8f4bcf3448991f469edffb80405 P 32500bcfa932410ab26501b83151acb3: Time spent starting tablet: real 0.020s user 0.006s sys 0.002s
I20260804 14:53:21.102026 2260 tablet_bootstrap.cc:492] T 5618b8f4bcf3448991f469edffb80405 P 3b87eee3311049e3873f76a8b7abd9ee: No bootstrap required, opened a new log
I20260804 14:53:21.102270 2260 ts_tablet_manager.cc:1397] T 5618b8f4bcf3448991f469edffb80405 P 3b87eee3311049e3873f76a8b7abd9ee: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260804 14:53:21.103020 2250 raft_consensus.cc:348] T 5618b8f4bcf3448991f469edffb80405 P e2ba1c7294da4cb89d31ce0113c187f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } } peers { permanent_uuid: "32500bcfa932410ab26501b83151acb3" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 35213 } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } }
I20260804 14:53:21.103389 2250 raft_consensus.cc:729] T 5618b8f4bcf3448991f469edffb80405 P e2ba1c7294da4cb89d31ce0113c187f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2ba1c7294da4cb89d31ce0113c187f4, State: Initialized, Role: FOLLOWER
I20260804 14:53:21.103750 2250 consensus_queue.cc:260] T 5618b8f4bcf3448991f469edffb80405 P e2ba1c7294da4cb89d31ce0113c187f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } } peers { permanent_uuid: "32500bcfa932410ab26501b83151acb3" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 35213 } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } }
I20260804 14:53:21.104626 1843 heartbeater.cc:503] Master 127.24.6.190:36209 was elected leader, sending a full tablet report...
I20260804 14:53:21.106434 2250 ts_tablet_manager.cc:1428] T 5618b8f4bcf3448991f469edffb80405 P e2ba1c7294da4cb89d31ce0113c187f4: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20260804 14:53:21.106657 24602 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20260804 14:53:21.107460 2216 heartbeater.cc:503] Master 127.24.6.190:36209 was elected leader, sending a full tablet report...
I20260804 14:53:21.107329 2260 raft_consensus.cc:348] T 5618b8f4bcf3448991f469edffb80405 P 3b87eee3311049e3873f76a8b7abd9ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } } peers { permanent_uuid: "32500bcfa932410ab26501b83151acb3" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 35213 } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } }
I20260804 14:53:21.107683 2260 raft_consensus.cc:729] T 5618b8f4bcf3448991f469edffb80405 P 3b87eee3311049e3873f76a8b7abd9ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b87eee3311049e3873f76a8b7abd9ee, State: Initialized, Role: FOLLOWER
I20260804 14:53:21.108083 2260 consensus_queue.cc:260] T 5618b8f4bcf3448991f469edffb80405 P 3b87eee3311049e3873f76a8b7abd9ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } } peers { permanent_uuid: "32500bcfa932410ab26501b83151acb3" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 35213 } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } }
I20260804 14:53:21.108851 2023 heartbeater.cc:503] Master 127.24.6.190:36209 was elected leader, sending a full tablet report...
I20260804 14:53:21.109522 2260 ts_tablet_manager.cc:1428] T 5618b8f4bcf3448991f469edffb80405 P 3b87eee3311049e3873f76a8b7abd9ee: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260804 14:53:21.109972 24602 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 6828849eb09e4f7d8434388177605134 to finish bootstrapping
I20260804 14:53:21.118009 24602 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 7bea20298f68472fbbb0a631dc011bc3 to finish bootstrapping
I20260804 14:53:21.125723 24602 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e2ba1c7294da4cb89d31ce0113c187f4 to finish bootstrapping
I20260804 14:53:21.132643 24602 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 3b87eee3311049e3873f76a8b7abd9ee to finish bootstrapping
W20260804 14:53:21.135275 1650 tablet.cc:2369] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260804 14:53:21.139675 24602 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 32500bcfa932410ab26501b83151acb3 to finish bootstrapping
I20260804 14:53:21.151674 1578 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5618b8f4bcf3448991f469edffb80405"
dest_uuid: "7bea20298f68472fbbb0a631dc011bc3"
from {username='slave'} at 127.0.0.1:54456
I20260804 14:53:21.151944 1578 raft_consensus.cc:482] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260804 14:53:21.152112 1578 raft_consensus.cc:3037] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:21.153434 1578 raft_consensus.cc:504] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } } peers { permanent_uuid: "32500bcfa932410ab26501b83151acb3" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 35213 } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } }
W20260804 14:53:21.153576 1446 tablet.cc:2369] T 5618b8f4bcf3448991f469edffb80405 P 6828849eb09e4f7d8434388177605134: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260804 14:53:21.154909 1578 leader_election.cc:288] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3 [CANDIDATE]: Term 1 election: Requested vote from peers 6828849eb09e4f7d8434388177605134 (127.24.6.129:34733), 3b87eee3311049e3873f76a8b7abd9ee (127.24.6.132:34633), 32500bcfa932410ab26501b83151acb3 (127.24.6.133:35213), e2ba1c7294da4cb89d31ce0113c187f4 (127.24.6.131:42643)
I20260804 14:53:21.160636 1385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5618b8f4bcf3448991f469edffb80405" candidate_uuid: "7bea20298f68472fbbb0a631dc011bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6828849eb09e4f7d8434388177605134"
I20260804 14:53:21.160908 1385 raft_consensus.cc:3037] T 5618b8f4bcf3448991f469edffb80405 P 6828849eb09e4f7d8434388177605134 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:21.162176 1385 raft_consensus.cc:2454] T 5618b8f4bcf3448991f469edffb80405 P 6828849eb09e4f7d8434388177605134 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bea20298f68472fbbb0a631dc011bc3 in term 1.
I20260804 14:53:21.163892 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5618b8f4bcf3448991f469edffb80405" candidate_uuid: "7bea20298f68472fbbb0a631dc011bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "32500bcfa932410ab26501b83151acb3"
I20260804 14:53:21.164229 2143 raft_consensus.cc:3037] T 5618b8f4bcf3448991f469edffb80405 P 32500bcfa932410ab26501b83151acb3 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:21.165316 1961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5618b8f4bcf3448991f469edffb80405" candidate_uuid: "7bea20298f68472fbbb0a631dc011bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3b87eee3311049e3873f76a8b7abd9ee"
I20260804 14:53:21.165315 1777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5618b8f4bcf3448991f469edffb80405" candidate_uuid: "7bea20298f68472fbbb0a631dc011bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e2ba1c7294da4cb89d31ce0113c187f4"
I20260804 14:53:21.165570 1961 raft_consensus.cc:3037] T 5618b8f4bcf3448991f469edffb80405 P 3b87eee3311049e3873f76a8b7abd9ee [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:21.165558 1777 raft_consensus.cc:3037] T 5618b8f4bcf3448991f469edffb80405 P e2ba1c7294da4cb89d31ce0113c187f4 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:21.166051 2143 raft_consensus.cc:2454] T 5618b8f4bcf3448991f469edffb80405 P 32500bcfa932410ab26501b83151acb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bea20298f68472fbbb0a631dc011bc3 in term 1.
I20260804 14:53:21.166584 1474 leader_election.cc:302] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 32500bcfa932410ab26501b83151acb3, 6828849eb09e4f7d8434388177605134, 7bea20298f68472fbbb0a631dc011bc3; no voters:
I20260804 14:53:21.166898 1777 raft_consensus.cc:2454] T 5618b8f4bcf3448991f469edffb80405 P e2ba1c7294da4cb89d31ce0113c187f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bea20298f68472fbbb0a631dc011bc3 in term 1.
I20260804 14:53:21.166898 1961 raft_consensus.cc:2454] T 5618b8f4bcf3448991f469edffb80405 P 3b87eee3311049e3873f76a8b7abd9ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bea20298f68472fbbb0a631dc011bc3 in term 1.
I20260804 14:53:21.166958 2259 raft_consensus.cc:2781] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:21.167296 2259 raft_consensus.cc:686] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3 [term 1 LEADER]: Becoming Leader. State: Replica: 7bea20298f68472fbbb0a631dc011bc3, State: Running, Role: LEADER
I20260804 14:53:21.167723 2259 consensus_queue.cc:237] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } } peers { permanent_uuid: "32500bcfa932410ab26501b83151acb3" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 35213 } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } }
I20260804 14:53:21.171841 1181 catalog_manager.cc:5495] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bea20298f68472fbbb0a631dc011bc3 (127.24.6.130). New cstate: current_term: 1 leader_uuid: "7bea20298f68472fbbb0a631dc011bc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32500bcfa932410ab26501b83151acb3" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 35213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } health_report { overall_health: HEALTHY } } }
W20260804 14:53:21.261941 2217 tablet.cc:2369] T 5618b8f4bcf3448991f469edffb80405 P 32500bcfa932410ab26501b83151acb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260804 14:53:21.303287 2024 tablet.cc:2369] T 5618b8f4bcf3448991f469edffb80405 P 3b87eee3311049e3873f76a8b7abd9ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260804 14:53:21.349309 1844 tablet.cc:2369] T 5618b8f4bcf3448991f469edffb80405 P e2ba1c7294da4cb89d31ce0113c187f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260804 14:53:21.693154 2280 consensus_queue.cc:1035] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32500bcfa932410ab26501b83151acb3" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 35213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260804 14:53:21.727419 2280 consensus_queue.cc:1035] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260804 14:53:21.734462 2286 consensus_queue.cc:1035] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260804 14:53:21.735273 2288 consensus_queue.cc:1035] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260804 14:53:21.739529 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 1145
W20260804 14:53:21.755102 1864 connection.cc:535] client connection to 127.24.6.190:36209 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260804 14:53:21.755527 2023 heartbeater.cc:644] Failed to heartbeat to 127.24.6.190:36209 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260804 14:53:21.755874 1843 heartbeater.cc:644] Failed to heartbeat to 127.24.6.190:36209 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.24.6.190:36209 (error 108)
I20260804 14:53:21.756250 24602 raft_consensus_election-itest.cc:666] Removing tserver with uuid 32500bcfa932410ab26501b83151acb3
I20260804 14:53:21.757942 1578 consensus_queue.cc:237] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } }
I20260804 14:53:21.759788 1385 raft_consensus.cc:1261] T 5618b8f4bcf3448991f469edffb80405 P 6828849eb09e4f7d8434388177605134 [term 1 FOLLOWER]: Refusing update from remote peer 7bea20298f68472fbbb0a631dc011bc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260804 14:53:21.759908 1961 raft_consensus.cc:1261] T 5618b8f4bcf3448991f469edffb80405 P 3b87eee3311049e3873f76a8b7abd9ee [term 1 FOLLOWER]: Refusing update from remote peer 7bea20298f68472fbbb0a631dc011bc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260804 14:53:21.760244 1777 raft_consensus.cc:1261] T 5618b8f4bcf3448991f469edffb80405 P e2ba1c7294da4cb89d31ce0113c187f4 [term 1 FOLLOWER]: Refusing update from remote peer 7bea20298f68472fbbb0a631dc011bc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260804 14:53:21.760330 2288 consensus_queue.cc:1035] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260804 14:53:21.760536 2289 consensus_queue.cc:1035] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260804 14:53:21.760687 2280 consensus_queue.cc:1035] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260804 14:53:21.763629 2280 raft_consensus.cc:2932] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 32500bcfa932410ab26501b83151acb3 (127.24.6.133) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } } }
I20260804 14:53:21.764024 1777 raft_consensus.cc:2932] T 5618b8f4bcf3448991f469edffb80405 P e2ba1c7294da4cb89d31ce0113c187f4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 32500bcfa932410ab26501b83151acb3 (127.24.6.133) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } } }
I20260804 14:53:21.764204 1385 raft_consensus.cc:2932] T 5618b8f4bcf3448991f469edffb80405 P 6828849eb09e4f7d8434388177605134 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 32500bcfa932410ab26501b83151acb3 (127.24.6.133) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } } }
W20260804 14:53:21.768028 1647 heartbeater.cc:644] Failed to heartbeat to 127.24.6.190:36209 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.6.190:36209: connect: Connection refused (error 111)
W20260804 14:53:21.768149 1444 heartbeater.cc:644] Failed to heartbeat to 127.24.6.190:36209 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.6.190:36209: connect: Connection refused (error 111)
I20260804 14:53:21.768980 1961 raft_consensus.cc:2932] T 5618b8f4bcf3448991f469edffb80405 P 3b87eee3311049e3873f76a8b7abd9ee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 32500bcfa932410ab26501b83151acb3 (127.24.6.133) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } } }
I20260804 14:53:21.787367 24602 raft_consensus_election-itest.cc:676] Pausing 3 replicas...
I20260804 14:53:21.819999 24602 raft_consensus_election-itest.cc:684] Adding back Peer 32500bcfa932410ab26501b83151acb3 and expecting timeout...
I20260804 14:53:21.821570 1578 consensus_queue.cc:237] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } } peers { permanent_uuid: "32500bcfa932410ab26501b83151acb3" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 35213 } attrs { } }
I20260804 14:53:21.823377 2143 raft_consensus.cc:1261] T 5618b8f4bcf3448991f469edffb80405 P 32500bcfa932410ab26501b83151acb3 [term 1 FOLLOWER]: Refusing update from remote peer 7bea20298f68472fbbb0a631dc011bc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260804 14:53:21.823880 2286 consensus_queue.cc:1035] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32500bcfa932410ab26501b83151acb3" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 35213 } attrs { }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260804 14:53:21.825644 2143 raft_consensus.cc:858] T 5618b8f4bcf3448991f469edffb80405 P 32500bcfa932410ab26501b83151acb3 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } } peers { permanent_uuid: "32500bcfa932410ab26501b83151acb3" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 35213 } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } }.
Pending config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } }:
New config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } } peers { permanent_uuid: "32500bcfa932410ab26501b83151acb3" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 35213 } attrs { } }
W20260804 14:53:21.825733 2143 raft_consensus.cc:1565] T 5618b8f4bcf3448991f469edffb80405 P 32500bcfa932410ab26501b83151acb3 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 3' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } } peers { permanent_uuid: "32500bcfa932410ab26501b83151acb3" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 35213 } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } }.
Pending config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } }:
New config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } } peers { permanent_uuid: "32500bcfa932410ab26501b83151acb3" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 35213 } attrs { } }
I20260804 14:53:21.826251 2143 raft_consensus.cc:2932] T 5618b8f4bcf3448991f469edffb80405 P 32500bcfa932410ab26501b83151acb3 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 32500bcfa932410ab26501b83151acb3 (127.24.6.133) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } } }
I20260804 14:53:21.829428 2143 raft_consensus.cc:1467] T 5618b8f4bcf3448991f469edffb80405 P 32500bcfa932410ab26501b83151acb3 [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
W20260804 14:53:21.829602 2216 heartbeater.cc:644] Failed to heartbeat to 127.24.6.190:36209 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.6.190:36209: connect: Connection refused (error 111)
I20260804 14:53:21.921466 24602 raft_consensus_election-itest.cc:688] Timeout achieved.
I20260804 14:53:21.928360 24602 raft_consensus_election-itest.cc:699] Forcing Peer 7bea20298f68472fbbb0a631dc011bc3 to step down...
I20260804 14:53:21.929073 1578 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5618b8f4bcf3448991f469edffb80405"
dest_uuid: "7bea20298f68472fbbb0a631dc011bc3"
from {username='slave'} at 127.0.0.1:54456
I20260804 14:53:21.929445 1578 raft_consensus.cc:580] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3 [term 1 LEADER]: Received request to step down
I20260804 14:53:21.929530 1578 raft_consensus.cc:3032] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3 [term 1 LEADER]: Stepping down as leader of term 1
I20260804 14:53:21.929582 1578 raft_consensus.cc:729] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7bea20298f68472fbbb0a631dc011bc3, State: Running, Role: LEADER
I20260804 14:53:21.929754 1578 consensus_queue.cc:260] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } } peers { permanent_uuid: "32500bcfa932410ab26501b83151acb3" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 35213 } attrs { } }
I20260804 14:53:21.930035 1578 raft_consensus.cc:3037] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3 [term 1 FOLLOWER]: Advancing to term 2
I20260804 14:53:21.930588 24602 raft_consensus_election-itest.cc:705] Resuming Peer 6828849eb09e4f7d8434388177605134 ...
I20260804 14:53:21.931428 24602 raft_consensus_election-itest.cc:710] Trying to elect Peer 32500bcfa932410ab26501b83151acb3 ...
I20260804 14:53:21.932104 1385 raft_consensus.cc:1261] T 5618b8f4bcf3448991f469edffb80405 P 6828849eb09e4f7d8434388177605134 [term 1 FOLLOWER]: Refusing update from remote peer 7bea20298f68472fbbb0a631dc011bc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260804 14:53:21.932086 2143 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5618b8f4bcf3448991f469edffb80405"
dest_uuid: "32500bcfa932410ab26501b83151acb3"
from {username='slave'} at 127.0.0.1:54944
I20260804 14:53:21.932417 2143 raft_consensus.cc:482] T 5618b8f4bcf3448991f469edffb80405 P 32500bcfa932410ab26501b83151acb3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260804 14:53:21.932513 2143 raft_consensus.cc:3037] T 5618b8f4bcf3448991f469edffb80405 P 32500bcfa932410ab26501b83151acb3 [term 1 FOLLOWER]: Advancing to term 2
I20260804 14:53:21.933703 2143 raft_consensus.cc:504] T 5618b8f4bcf3448991f469edffb80405 P 32500bcfa932410ab26501b83151acb3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } } peers { permanent_uuid: "32500bcfa932410ab26501b83151acb3" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 35213 } attrs { } }
I20260804 14:53:21.935062 2143 leader_election.cc:288] T 5618b8f4bcf3448991f469edffb80405 P 32500bcfa932410ab26501b83151acb3 [CANDIDATE]: Term 2 election: Requested vote from peers 6828849eb09e4f7d8434388177605134 (127.24.6.129:34733), 3b87eee3311049e3873f76a8b7abd9ee (127.24.6.132:34633), e2ba1c7294da4cb89d31ce0113c187f4 (127.24.6.131:42643), 7bea20298f68472fbbb0a631dc011bc3 (127.24.6.130:39539)
I20260804 14:53:21.940047 1385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5618b8f4bcf3448991f469edffb80405" candidate_uuid: "32500bcfa932410ab26501b83151acb3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6828849eb09e4f7d8434388177605134"
I20260804 14:53:21.940237 1385 raft_consensus.cc:1799] T 5618b8f4bcf3448991f469edffb80405 P 6828849eb09e4f7d8434388177605134 [term 1 FOLLOWER]: Handling vote request from an unknown peer 32500bcfa932410ab26501b83151acb3
I20260804 14:53:21.940306 1385 raft_consensus.cc:3037] T 5618b8f4bcf3448991f469edffb80405 P 6828849eb09e4f7d8434388177605134 [term 1 FOLLOWER]: Advancing to term 2
I20260804 14:53:21.940199 24602 cluster_itest_util.cc:256] Not converged past 4 yet: 1.2 1.3 1.3
I20260804 14:53:21.940495 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5618b8f4bcf3448991f469edffb80405" candidate_uuid: "32500bcfa932410ab26501b83151acb3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "7bea20298f68472fbbb0a631dc011bc3"
I20260804 14:53:21.941489 1385 raft_consensus.cc:2454] T 5618b8f4bcf3448991f469edffb80405 P 6828849eb09e4f7d8434388177605134 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32500bcfa932410ab26501b83151acb3 in term 2.
I20260804 14:53:21.941586 1578 raft_consensus.cc:2454] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32500bcfa932410ab26501b83151acb3 in term 2.
I20260804 14:53:21.942016 2050 leader_election.cc:302] T 5618b8f4bcf3448991f469edffb80405 P 32500bcfa932410ab26501b83151acb3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 32500bcfa932410ab26501b83151acb3, 6828849eb09e4f7d8434388177605134, 7bea20298f68472fbbb0a631dc011bc3; no voters:
I20260804 14:53:21.942317 2281 raft_consensus.cc:2781] T 5618b8f4bcf3448991f469edffb80405 P 32500bcfa932410ab26501b83151acb3 [term 2 FOLLOWER]: Leader election won for term 2
I20260804 14:53:21.942595 2281 raft_consensus.cc:686] T 5618b8f4bcf3448991f469edffb80405 P 32500bcfa932410ab26501b83151acb3 [term 2 LEADER]: Becoming Leader. State: Replica: 32500bcfa932410ab26501b83151acb3, State: Running, Role: LEADER
I20260804 14:53:21.942966 2281 consensus_queue.cc:237] T 5618b8f4bcf3448991f469edffb80405 P 32500bcfa932410ab26501b83151acb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } } peers { permanent_uuid: "32500bcfa932410ab26501b83151acb3" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 35213 } attrs { } }
I20260804 14:53:22.044224 24602 cluster_itest_util.cc:256] Not converged past 4 yet: 1.2 1.3 2.4
I20260804 14:53:22.247741 24602 cluster_itest_util.cc:256] Not converged past 4 yet: 1.2 1.3 2.4
I20260804 14:53:22.362172 1385 raft_consensus.cc:1261] T 5618b8f4bcf3448991f469edffb80405 P 6828849eb09e4f7d8434388177605134 [term 2 FOLLOWER]: Refusing update from remote peer 32500bcfa932410ab26501b83151acb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260804 14:53:22.362839 2281 consensus_queue.cc:1035] T 5618b8f4bcf3448991f469edffb80405 P 32500bcfa932410ab26501b83151acb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260804 14:53:22.463047 1578 raft_consensus.cc:1261] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3 [term 2 FOLLOWER]: Refusing update from remote peer 32500bcfa932410ab26501b83151acb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260804 14:53:22.463915 2281 consensus_queue.cc:1035] T 5618b8f4bcf3448991f469edffb80405 P 32500bcfa932410ab26501b83151acb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260804 14:53:22.466704 2328 raft_consensus.cc:2932] T 5618b8f4bcf3448991f469edffb80405 P 32500bcfa932410ab26501b83151acb3 [term 2 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 32500bcfa932410ab26501b83151acb3 (127.24.6.133) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } } peers { permanent_uuid: "32500bcfa932410ab26501b83151acb3" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 35213 } attrs { } } }
I20260804 14:53:22.467314 1578 raft_consensus.cc:2932] T 5618b8f4bcf3448991f469edffb80405 P 7bea20298f68472fbbb0a631dc011bc3 [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 32500bcfa932410ab26501b83151acb3 (127.24.6.133) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } } peers { permanent_uuid: "32500bcfa932410ab26501b83151acb3" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 35213 } attrs { } } }
W20260804 14:53:22.468710 1578 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:54456 (request call id 54) took 647 ms (client timeout 100 ms). Trace:
W20260804 14:53:22.468806 1578 rpcz_store.cc:269] 0804 14:53:21.820945 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:53:21.821039 (+ 94us) service_pool.cc:227] Handling call
0804 14:53:21.822321 (+ 1282us) log.cc:842] Serialized 600 byte log entry
0804 14:53:22.468665 (+646344us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":433,"raft.run_cpu_time_us":513,"raft.run_wall_time_us":515,"spinlock_wait_cycles":131584}
I20260804 14:53:22.469475 1385 raft_consensus.cc:2932] T 5618b8f4bcf3448991f469edffb80405 P 6828849eb09e4f7d8434388177605134 [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 32500bcfa932410ab26501b83151acb3 (127.24.6.133) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } } peers { permanent_uuid: "32500bcfa932410ab26501b83151acb3" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 35213 } attrs { } } }
I20260804 14:53:22.551304 24602 raft_consensus_election-itest.cc:716] Resuming remaining nodes...
I20260804 14:53:22.553740 1961 raft_consensus.cc:1261] T 5618b8f4bcf3448991f469edffb80405 P 3b87eee3311049e3873f76a8b7abd9ee [term 1 FOLLOWER]: Refusing update from remote peer 7bea20298f68472fbbb0a631dc011bc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260804 14:53:22.553903 1777 raft_consensus.cc:1261] T 5618b8f4bcf3448991f469edffb80405 P e2ba1c7294da4cb89d31ce0113c187f4 [term 1 FOLLOWER]: Refusing update from remote peer 7bea20298f68472fbbb0a631dc011bc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260804 14:53:22.559333 1777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5618b8f4bcf3448991f469edffb80405" candidate_uuid: "32500bcfa932410ab26501b83151acb3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e2ba1c7294da4cb89d31ce0113c187f4"
I20260804 14:53:22.559495 1777 raft_consensus.cc:1799] T 5618b8f4bcf3448991f469edffb80405 P e2ba1c7294da4cb89d31ce0113c187f4 [term 1 FOLLOWER]: Handling vote request from an unknown peer 32500bcfa932410ab26501b83151acb3
I20260804 14:53:22.559551 1777 raft_consensus.cc:3037] T 5618b8f4bcf3448991f469edffb80405 P e2ba1c7294da4cb89d31ce0113c187f4 [term 1 FOLLOWER]: Advancing to term 2
I20260804 14:53:22.560712 1777 raft_consensus.cc:2454] T 5618b8f4bcf3448991f469edffb80405 P e2ba1c7294da4cb89d31ce0113c187f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32500bcfa932410ab26501b83151acb3 in term 2.
I20260804 14:53:22.561262 1961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5618b8f4bcf3448991f469edffb80405" candidate_uuid: "32500bcfa932410ab26501b83151acb3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3b87eee3311049e3873f76a8b7abd9ee"
I20260804 14:53:22.561393 1961 raft_consensus.cc:1799] T 5618b8f4bcf3448991f469edffb80405 P 3b87eee3311049e3873f76a8b7abd9ee [term 1 FOLLOWER]: Handling vote request from an unknown peer 32500bcfa932410ab26501b83151acb3
I20260804 14:53:22.561465 1961 raft_consensus.cc:3037] T 5618b8f4bcf3448991f469edffb80405 P 3b87eee3311049e3873f76a8b7abd9ee [term 1 FOLLOWER]: Advancing to term 2
I20260804 14:53:22.562050 1777 raft_consensus.cc:1261] T 5618b8f4bcf3448991f469edffb80405 P e2ba1c7294da4cb89d31ce0113c187f4 [term 2 FOLLOWER]: Refusing update from remote peer 32500bcfa932410ab26501b83151acb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260804 14:53:22.562538 1961 raft_consensus.cc:2454] T 5618b8f4bcf3448991f469edffb80405 P 3b87eee3311049e3873f76a8b7abd9ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32500bcfa932410ab26501b83151acb3 in term 2.
I20260804 14:53:22.562518 2281 consensus_queue.cc:1035] T 5618b8f4bcf3448991f469edffb80405 P 32500bcfa932410ab26501b83151acb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260804 14:53:22.562837 1960 raft_consensus.cc:1261] T 5618b8f4bcf3448991f469edffb80405 P 3b87eee3311049e3873f76a8b7abd9ee [term 2 FOLLOWER]: Refusing update from remote peer 32500bcfa932410ab26501b83151acb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260804 14:53:22.563970 2330 consensus_queue.cc:1035] T 5618b8f4bcf3448991f469edffb80405 P 32500bcfa932410ab26501b83151acb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260804 14:53:22.566727 1960 raft_consensus.cc:2932] T 5618b8f4bcf3448991f469edffb80405 P 3b87eee3311049e3873f76a8b7abd9ee [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 32500bcfa932410ab26501b83151acb3 (127.24.6.133) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } } peers { permanent_uuid: "32500bcfa932410ab26501b83151acb3" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 35213 } attrs { } } }
I20260804 14:53:22.567806 1777 raft_consensus.cc:2932] T 5618b8f4bcf3448991f469edffb80405 P e2ba1c7294da4cb89d31ce0113c187f4 [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 32500bcfa932410ab26501b83151acb3 (127.24.6.133) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6828849eb09e4f7d8434388177605134" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 34733 } } peers { permanent_uuid: "3b87eee3311049e3873f76a8b7abd9ee" member_type: VOTER last_known_addr { host: "127.24.6.132" port: 34633 } } peers { permanent_uuid: "e2ba1c7294da4cb89d31ce0113c187f4" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 42643 } } peers { permanent_uuid: "7bea20298f68472fbbb0a631dc011bc3" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 39539 } } peers { permanent_uuid: "32500bcfa932410ab26501b83151acb3" member_type: VOTER last_known_addr { host: "127.24.6.133" port: 35213 } attrs { } } }
I20260804 14:53:22.572841 2339 mvcc.cc:205] Tried to move back new op lower bound from 7314862909683613696 to 7314862907162857472. Current Snapshot: MvccSnapshot[applied={T|T < 7314862909683613696}]
I20260804 14:53:22.579705 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 1272
I20260804 14:53:22.599377 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 1451
I20260804 14:53:22.614935 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 1654
I20260804 14:53:22.629210 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 1847
I20260804 14:53:22.643669 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 2028
2026-08-04T14:53:22Z chronyd exiting
[ OK ] RaftConsensusElectionITest.ElectPendingVoter (3054 ms)
[ RUN ] RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy
I20260804 14:53:22.673295 24602 test_util.cc:268] Using random seed: -788938103
I20260804 14:53:22.674232 24602 ts_itest-base.cc:112] Starting cluster with:
I20260804 14:53:22.674292 24602 ts_itest-base.cc:113] --------------
I20260804 14:53:22.674314 24602 ts_itest-base.cc:114] 3 tablet servers
I20260804 14:53:22.674342 24602 ts_itest-base.cc:115] 3 replicas per TS
I20260804 14:53:22.674367 24602 ts_itest-base.cc:116] --------------
2026-08-04T14:53:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-04T14:53:22Z Disabled control of system clock
I20260804 14:53:22.686430 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.6.190:33693
--webserver_interface=127.24.6.190
--webserver_port=0
--builtin_ntp_servers=127.24.6.148:36237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.6.190:33693
--master_add_server_when_underreplicated=false
--raft_prepare_replacement_before_eviction=false with env {}
W20260804 14:53:22.810959 2370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:22.811419 2370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:22.811501 2370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:22.815857 2370 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260804 14:53:22.815991 2370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260804 14:53:22.816025 2370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:22.816064 2370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260804 14:53:22.816100 2370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260804 14:53:22.822142 2370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:36237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
--master_add_server_when_underreplicated=false
--ipki_ca_key_size=768
--master_addresses=127.24.6.190:33693
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.6.190:33693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.24.6.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.2370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:22.823601 2370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:22.825194 2370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:22.834929 2375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:22.834947 2376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:22.834954 2378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:22.835235 2370 server_base.cc:1034] running on GCE node
I20260804 14:53:22.836054 2370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:22.837400 2370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:22.838680 2370 hybrid_clock.cc:648] HybridClock initialized: now 1785855202838654 us; error 43 us; skew 500 ppm
I20260804 14:53:22.841223 2370 webserver.cc:460] Webserver started at http://127.24.6.190:42645/ using document root <none> and password file <none>
I20260804 14:53:22.841936 2370 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:22.842059 2370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:22.842337 2370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:22.844760 2370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "2f5202c683d8428da3a7273ca55e7827"
format_stamp: "Formatted at 2026-08-04 14:53:22 on dist-test-slave-92m1"
I20260804 14:53:22.845434 2370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "2f5202c683d8428da3a7273ca55e7827"
format_stamp: "Formatted at 2026-08-04 14:53:22 on dist-test-slave-92m1"
I20260804 14:53:22.849737 2370 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260804 14:53:22.852612 2386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:22.854063 2370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260804 14:53:22.854225 2370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2f5202c683d8428da3a7273ca55e7827"
format_stamp: "Formatted at 2026-08-04 14:53:22 on dist-test-slave-92m1"
I20260804 14:53:22.854359 2370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:22.873237 2370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:22.874315 2370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:22.874558 2370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:22.883313 2370 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.190:33693
I20260804 14:53:22.883353 2455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.190:33693 every 8 connection(s)
I20260804 14:53:22.884538 2370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260804 14:53:22.888398 2456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:22.894176 2456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827: Bootstrap starting.
I20260804 14:53:22.894685 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 2370
I20260804 14:53:22.894857 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260804 14:53:22.897326 2456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:22.898140 2456 log.cc:824] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:22.900384 2456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827: No bootstrap required, opened a new log
I20260804 14:53:22.904676 2456 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5202c683d8428da3a7273ca55e7827" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 33693 } }
I20260804 14:53:22.904973 2456 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:53:22.905071 2456 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f5202c683d8428da3a7273ca55e7827, State: Initialized, Role: FOLLOWER
I20260804 14:53:22.905645 2456 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5202c683d8428da3a7273ca55e7827" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 33693 } }
I20260804 14:53:22.905821 2456 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260804 14:53:22.905880 2456 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260804 14:53:22.905978 2456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:22.907004 2456 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5202c683d8428da3a7273ca55e7827" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 33693 } }
I20260804 14:53:22.907434 2456 leader_election.cc:302] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f5202c683d8428da3a7273ca55e7827; no voters:
I20260804 14:53:22.907820 2456 leader_election.cc:288] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260804 14:53:22.907972 2464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:22.908265 2464 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [term 1 LEADER]: Becoming Leader. State: Replica: 2f5202c683d8428da3a7273ca55e7827, State: Running, Role: LEADER
I20260804 14:53:22.908633 2464 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5202c683d8428da3a7273ca55e7827" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 33693 } }
I20260804 14:53:22.909314 2456 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [sys.catalog]: configured and running, proceeding with master startup.
I20260804 14:53:22.910741 2467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f5202c683d8428da3a7273ca55e7827. Latest consensus state: current_term: 1 leader_uuid: "2f5202c683d8428da3a7273ca55e7827" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5202c683d8428da3a7273ca55e7827" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 33693 } } }
I20260804 14:53:22.910902 2467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [sys.catalog]: This master's current role is: LEADER
I20260804 14:53:22.911051 2466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f5202c683d8428da3a7273ca55e7827" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5202c683d8428da3a7273ca55e7827" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 33693 } } }
I20260804 14:53:22.911242 2466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [sys.catalog]: This master's current role is: LEADER
I20260804 14:53:22.911955 2479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260804 14:53:22.915068 2479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260804 14:53:22.921422 2479 catalog_manager.cc:1347] Generated new cluster ID: 68b0e8e5dd2b4878a25d5705a60e9fd0
I20260804 14:53:22.921536 2479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260804 14:53:22.942286 2479 catalog_manager.cc:1370] Generated new certificate authority record
I20260804 14:53:22.943444 2479 catalog_manager.cc:1504] Loading token signing keys...
I20260804 14:53:22.950379 2479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827: Generated new TSK 0
I20260804 14:53:22.951174 2479 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260804 14:53:22.954663 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.129:0
--local_ip_for_outbound_sockets=127.24.6.129
--webserver_interface=127.24.6.129
--webserver_port=0
--tserver_master_addrs=127.24.6.190:33693
--builtin_ntp_servers=127.24.6.148:36237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
W20260804 14:53:23.092851 2491 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20260804 14:53:23.093222 2491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:23.093245 2491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:23.093279 2491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:23.097595 2491 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20260804 14:53:23.097682 2491 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20260804 14:53:23.097707 2491 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260804 14:53:23.097731 2491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:23.097783 2491 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20260804 14:53:23.097802 2491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.129
I20260804 14:53:23.102828 2491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:36237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.24.6.129
--webserver_port=0
--tserver_master_addrs=127.24.6.190:33693
--never_fsync=true
--heap_profile_path=/tmp/kudu.2491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.129
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:23.104317 2491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:23.105734 2491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:23.116824 2501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:23.116863 2499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:23.117120 2491 server_base.cc:1034] running on GCE node
W20260804 14:53:23.116837 2498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:23.117723 2491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:23.118479 2491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:23.119683 2491 hybrid_clock.cc:648] HybridClock initialized: now 1785855203119665 us; error 33 us; skew 500 ppm
I20260804 14:53:23.121829 2491 webserver.cc:460] Webserver started at http://127.24.6.129:43463/ using document root <none> and password file <none>
I20260804 14:53:23.122501 2491 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:23.122606 2491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:23.122856 2491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:23.125124 2491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "9a7053094ff74e82b9a176d1db50adca"
format_stamp: "Formatted at 2026-08-04 14:53:23 on dist-test-slave-92m1"
I20260804 14:53:23.125788 2491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "9a7053094ff74e82b9a176d1db50adca"
format_stamp: "Formatted at 2026-08-04 14:53:23 on dist-test-slave-92m1"
I20260804 14:53:23.130322 2491 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260804 14:53:23.133271 2508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:23.134701 2491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260804 14:53:23.134868 2491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "9a7053094ff74e82b9a176d1db50adca"
format_stamp: "Formatted at 2026-08-04 14:53:23 on dist-test-slave-92m1"
I20260804 14:53:23.135008 2491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:23.148607 2491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:23.149638 2491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:23.149899 2491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:23.150636 2491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:53:23.151800 2491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:53:23.151898 2491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:23.152004 2491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:53:23.152074 2491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:23.163182 2491 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.129:40315
I20260804 14:53:23.163232 2667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.129:40315 every 8 connection(s)
I20260804 14:53:23.164470 2491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260804 14:53:23.174787 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 2491
I20260804 14:53:23.174964 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260804 14:53:23.177018 2669 heartbeater.cc:347] Connected to a master server at 127.24.6.190:33693
I20260804 14:53:23.177398 2669 heartbeater.cc:464] Registering TS with master...
I20260804 14:53:23.177865 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.130:0
--local_ip_for_outbound_sockets=127.24.6.130
--webserver_interface=127.24.6.130
--webserver_port=0
--tserver_master_addrs=127.24.6.190:33693
--builtin_ntp_servers=127.24.6.148:36237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20260804 14:53:23.178089 2669 heartbeater.cc:511] Master 127.24.6.190:33693 requested a full tablet report, sending...
I20260804 14:53:23.180186 2410 ts_manager.cc:194] Registered new tserver with Master: 9a7053094ff74e82b9a176d1db50adca (127.24.6.129:40315)
I20260804 14:53:23.182169 2410 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.129:50409
W20260804 14:53:23.312834 2674 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20260804 14:53:23.313192 2674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:23.313217 2674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:23.313256 2674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:23.318131 2674 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20260804 14:53:23.318214 2674 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20260804 14:53:23.318239 2674 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260804 14:53:23.318262 2674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:23.318315 2674 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20260804 14:53:23.318334 2674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.130
I20260804 14:53:23.323684 2674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:36237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.24.6.130
--webserver_port=0
--tserver_master_addrs=127.24.6.190:33693
--never_fsync=true
--heap_profile_path=/tmp/kudu.2674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.130
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:23.324905 2674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:23.326460 2674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:23.338071 2685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:23.338222 2681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:23.338282 2674 server_base.cc:1034] running on GCE node
W20260804 14:53:23.338222 2682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:23.338995 2674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:23.339934 2674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:23.341161 2674 hybrid_clock.cc:648] HybridClock initialized: now 1785855203341135 us; error 39 us; skew 500 ppm
I20260804 14:53:23.343667 2674 webserver.cc:460] Webserver started at http://127.24.6.130:42173/ using document root <none> and password file <none>
I20260804 14:53:23.344398 2674 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:23.344475 2674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:23.344728 2674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:23.346920 2674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "dc0d993490e247b9a6aa70eedc63eb06"
format_stamp: "Formatted at 2026-08-04 14:53:23 on dist-test-slave-92m1"
I20260804 14:53:23.347702 2674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "dc0d993490e247b9a6aa70eedc63eb06"
format_stamp: "Formatted at 2026-08-04 14:53:23 on dist-test-slave-92m1"
I20260804 14:53:23.352051 2674 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260804 14:53:23.355190 2695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:23.356730 2674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260804 14:53:23.356890 2674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "dc0d993490e247b9a6aa70eedc63eb06"
format_stamp: "Formatted at 2026-08-04 14:53:23 on dist-test-slave-92m1"
I20260804 14:53:23.357074 2674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:23.382864 2674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:23.383738 2674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:23.383961 2674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:23.384635 2674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:53:23.385675 2674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:53:23.385726 2674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:23.385799 2674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:53:23.385840 2674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:23.396857 2674 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.130:41505
I20260804 14:53:23.396883 2861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.130:41505 every 8 connection(s)
I20260804 14:53:23.397856 2674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260804 14:53:23.398289 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 2674
I20260804 14:53:23.398454 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260804 14:53:23.401122 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.131:0
--local_ip_for_outbound_sockets=127.24.6.131
--webserver_interface=127.24.6.131
--webserver_port=0
--tserver_master_addrs=127.24.6.190:33693
--builtin_ntp_servers=127.24.6.148:36237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20260804 14:53:23.412249 2862 heartbeater.cc:347] Connected to a master server at 127.24.6.190:33693
I20260804 14:53:23.412555 2862 heartbeater.cc:464] Registering TS with master...
I20260804 14:53:23.413100 2862 heartbeater.cc:511] Master 127.24.6.190:33693 requested a full tablet report, sending...
I20260804 14:53:23.414238 2410 ts_manager.cc:194] Registered new tserver with Master: dc0d993490e247b9a6aa70eedc63eb06 (127.24.6.130:41505)
I20260804 14:53:23.414865 2410 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.130:33575
W20260804 14:53:23.529608 2868 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20260804 14:53:23.529940 2868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:23.529961 2868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:23.530043 2868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:23.535027 2868 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20260804 14:53:23.535109 2868 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20260804 14:53:23.535133 2868 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260804 14:53:23.535156 2868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:23.535270 2868 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20260804 14:53:23.535296 2868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.131
I20260804 14:53:23.540827 2868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:36237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.24.6.131
--webserver_port=0
--tserver_master_addrs=127.24.6.190:33693
--never_fsync=true
--heap_profile_path=/tmp/kudu.2868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.131
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:23.542578 2868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:23.544947 2868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:23.557089 2874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:23.557089 2875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:23.557106 2878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:23.557516 2868 server_base.cc:1034] running on GCE node
I20260804 14:53:23.558032 2868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:23.558821 2868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:23.560079 2868 hybrid_clock.cc:648] HybridClock initialized: now 1785855203560078 us; error 53 us; skew 500 ppm
I20260804 14:53:23.562512 2868 webserver.cc:460] Webserver started at http://127.24.6.131:45865/ using document root <none> and password file <none>
I20260804 14:53:23.563161 2868 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:23.563313 2868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:23.563565 2868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:23.565671 2868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "422632429a694bde88205930ec518cc6"
format_stamp: "Formatted at 2026-08-04 14:53:23 on dist-test-slave-92m1"
I20260804 14:53:23.566355 2868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "422632429a694bde88205930ec518cc6"
format_stamp: "Formatted at 2026-08-04 14:53:23 on dist-test-slave-92m1"
I20260804 14:53:23.570748 2868 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260804 14:53:23.574149 2886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:23.575891 2868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260804 14:53:23.576032 2868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "422632429a694bde88205930ec518cc6"
format_stamp: "Formatted at 2026-08-04 14:53:23 on dist-test-slave-92m1"
I20260804 14:53:23.576151 2868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:23.594441 2868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:23.595184 2868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:23.595479 2868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:23.596091 2868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:53:23.597084 2868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:53:23.597132 2868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:23.597173 2868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:53:23.597189 2868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:23.607051 2868 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.131:41365
I20260804 14:53:23.607081 3049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.131:41365 every 8 connection(s)
I20260804 14:53:23.608112 2868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260804 14:53:23.610495 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 2868
I20260804 14:53:23.610636 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260804 14:53:23.620532 3050 heartbeater.cc:347] Connected to a master server at 127.24.6.190:33693
I20260804 14:53:23.620814 3050 heartbeater.cc:464] Registering TS with master...
I20260804 14:53:23.621299 3050 heartbeater.cc:511] Master 127.24.6.190:33693 requested a full tablet report, sending...
I20260804 14:53:23.622599 2410 ts_manager.cc:194] Registered new tserver with Master: 422632429a694bde88205930ec518cc6 (127.24.6.131:41365)
I20260804 14:53:23.623327 2410 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.131:40149
I20260804 14:53:23.625953 24602 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260804 14:53:23.637694 2410 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39620:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20260804 14:53:23.661314 2574 tablet_service.cc:1467] Processing CreateTablet for tablet e62bf4b11eab47f193f28d71e8bd8bdf (DEFAULT_TABLE table=TestTable [id=375975d53e964d7ea790eb3690c3fd2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:53:23.661321 2951 tablet_service.cc:1467] Processing CreateTablet for tablet e62bf4b11eab47f193f28d71e8bd8bdf (DEFAULT_TABLE table=TestTable [id=375975d53e964d7ea790eb3690c3fd2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:53:23.661322 2765 tablet_service.cc:1467] Processing CreateTablet for tablet e62bf4b11eab47f193f28d71e8bd8bdf (DEFAULT_TABLE table=TestTable [id=375975d53e964d7ea790eb3690c3fd2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:53:23.662536 2574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e62bf4b11eab47f193f28d71e8bd8bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:23.662539 2951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e62bf4b11eab47f193f28d71e8bd8bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:23.662539 2765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e62bf4b11eab47f193f28d71e8bd8bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:23.669340 3077 tablet_bootstrap.cc:492] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca: Bootstrap starting.
I20260804 14:53:23.670526 3079 tablet_bootstrap.cc:492] T e62bf4b11eab47f193f28d71e8bd8bdf P dc0d993490e247b9a6aa70eedc63eb06: Bootstrap starting.
I20260804 14:53:23.671591 3078 tablet_bootstrap.cc:492] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6: Bootstrap starting.
I20260804 14:53:23.671900 3077 tablet_bootstrap.cc:654] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:23.672696 3077 log.cc:824] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:23.672711 3079 tablet_bootstrap.cc:654] T e62bf4b11eab47f193f28d71e8bd8bdf P dc0d993490e247b9a6aa70eedc63eb06: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:23.673623 3079 log.cc:824] T e62bf4b11eab47f193f28d71e8bd8bdf P dc0d993490e247b9a6aa70eedc63eb06: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:23.673780 3078 tablet_bootstrap.cc:654] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:23.674603 3078 log.cc:824] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:23.674953 3077 tablet_bootstrap.cc:492] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca: No bootstrap required, opened a new log
I20260804 14:53:23.675191 3077 ts_tablet_manager.cc:1397] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260804 14:53:23.675755 3079 tablet_bootstrap.cc:492] T e62bf4b11eab47f193f28d71e8bd8bdf P dc0d993490e247b9a6aa70eedc63eb06: No bootstrap required, opened a new log
I20260804 14:53:23.675983 3079 ts_tablet_manager.cc:1397] T e62bf4b11eab47f193f28d71e8bd8bdf P dc0d993490e247b9a6aa70eedc63eb06: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260804 14:53:23.676930 3078 tablet_bootstrap.cc:492] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6: No bootstrap required, opened a new log
I20260804 14:53:23.677139 3078 ts_tablet_manager.cc:1397] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260804 14:53:23.680028 3079 raft_consensus.cc:348] T e62bf4b11eab47f193f28d71e8bd8bdf P dc0d993490e247b9a6aa70eedc63eb06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 } } peers { permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 } } peers { permanent_uuid: "9a7053094ff74e82b9a176d1db50adca" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40315 } }
I20260804 14:53:23.680189 3077 raft_consensus.cc:348] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 } } peers { permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 } } peers { permanent_uuid: "9a7053094ff74e82b9a176d1db50adca" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40315 } }
I20260804 14:53:23.680464 3079 raft_consensus.cc:374] T e62bf4b11eab47f193f28d71e8bd8bdf P dc0d993490e247b9a6aa70eedc63eb06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:53:23.680513 3077 raft_consensus.cc:374] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:53:23.680532 3079 raft_consensus.cc:729] T e62bf4b11eab47f193f28d71e8bd8bdf P dc0d993490e247b9a6aa70eedc63eb06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc0d993490e247b9a6aa70eedc63eb06, State: Initialized, Role: FOLLOWER
I20260804 14:53:23.680567 3077 raft_consensus.cc:729] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a7053094ff74e82b9a176d1db50adca, State: Initialized, Role: FOLLOWER
I20260804 14:53:23.681085 3079 consensus_queue.cc:260] T e62bf4b11eab47f193f28d71e8bd8bdf P dc0d993490e247b9a6aa70eedc63eb06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 } } peers { permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 } } peers { permanent_uuid: "9a7053094ff74e82b9a176d1db50adca" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40315 } }
I20260804 14:53:23.681095 3077 consensus_queue.cc:260] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 } } peers { permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 } } peers { permanent_uuid: "9a7053094ff74e82b9a176d1db50adca" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40315 } }
I20260804 14:53:23.681461 3085 raft_consensus.cc:482] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260804 14:53:23.681658 3085 raft_consensus.cc:504] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 } } peers { permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 } } peers { permanent_uuid: "9a7053094ff74e82b9a176d1db50adca" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40315 } }
I20260804 14:53:23.682265 3079 ts_tablet_manager.cc:1428] T e62bf4b11eab47f193f28d71e8bd8bdf P dc0d993490e247b9a6aa70eedc63eb06: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260804 14:53:23.682273 3077 ts_tablet_manager.cc:1428] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260804 14:53:23.682662 2862 heartbeater.cc:503] Master 127.24.6.190:33693 was elected leader, sending a full tablet report...
I20260804 14:53:23.682662 2669 heartbeater.cc:503] Master 127.24.6.190:33693 was elected leader, sending a full tablet report...
I20260804 14:53:23.683120 3078 raft_consensus.cc:348] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 } } peers { permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 } } peers { permanent_uuid: "9a7053094ff74e82b9a176d1db50adca" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40315 } }
I20260804 14:53:23.683439 3078 raft_consensus.cc:374] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:53:23.683512 3078 raft_consensus.cc:729] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 422632429a694bde88205930ec518cc6, State: Initialized, Role: FOLLOWER
I20260804 14:53:23.683955 3078 consensus_queue.cc:260] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 } } peers { permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 } } peers { permanent_uuid: "9a7053094ff74e82b9a176d1db50adca" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40315 } }
I20260804 14:53:23.685038 3078 ts_tablet_manager.cc:1428] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260804 14:53:23.685412 3085 leader_election.cc:288] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 422632429a694bde88205930ec518cc6 (127.24.6.131:41365), dc0d993490e247b9a6aa70eedc63eb06 (127.24.6.130:41505)
I20260804 14:53:23.685576 3050 heartbeater.cc:503] Master 127.24.6.190:33693 was elected leader, sending a full tablet report...
I20260804 14:53:23.690666 2982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e62bf4b11eab47f193f28d71e8bd8bdf" candidate_uuid: "9a7053094ff74e82b9a176d1db50adca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "422632429a694bde88205930ec518cc6" is_pre_election: true
I20260804 14:53:23.691035 2982 raft_consensus.cc:2454] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a7053094ff74e82b9a176d1db50adca in term 0.
I20260804 14:53:23.691357 2791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e62bf4b11eab47f193f28d71e8bd8bdf" candidate_uuid: "9a7053094ff74e82b9a176d1db50adca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc0d993490e247b9a6aa70eedc63eb06" is_pre_election: true
I20260804 14:53:23.691675 2791 raft_consensus.cc:2454] T e62bf4b11eab47f193f28d71e8bd8bdf P dc0d993490e247b9a6aa70eedc63eb06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a7053094ff74e82b9a176d1db50adca in term 0.
I20260804 14:53:23.691636 2509 leader_election.cc:302] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 422632429a694bde88205930ec518cc6, 9a7053094ff74e82b9a176d1db50adca; no voters:
I20260804 14:53:23.692018 3086 raft_consensus.cc:2781] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260804 14:53:23.692129 3086 raft_consensus.cc:482] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260804 14:53:23.692191 3086 raft_consensus.cc:3037] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:23.693250 3086 raft_consensus.cc:504] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 } } peers { permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 } } peers { permanent_uuid: "9a7053094ff74e82b9a176d1db50adca" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40315 } }
I20260804 14:53:23.693591 3086 leader_election.cc:288] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [CANDIDATE]: Term 1 election: Requested vote from peers 422632429a694bde88205930ec518cc6 (127.24.6.131:41365), dc0d993490e247b9a6aa70eedc63eb06 (127.24.6.130:41505)
I20260804 14:53:23.694021 2791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e62bf4b11eab47f193f28d71e8bd8bdf" candidate_uuid: "9a7053094ff74e82b9a176d1db50adca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc0d993490e247b9a6aa70eedc63eb06"
I20260804 14:53:23.694036 2982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e62bf4b11eab47f193f28d71e8bd8bdf" candidate_uuid: "9a7053094ff74e82b9a176d1db50adca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "422632429a694bde88205930ec518cc6"
I20260804 14:53:23.694134 2791 raft_consensus.cc:3037] T e62bf4b11eab47f193f28d71e8bd8bdf P dc0d993490e247b9a6aa70eedc63eb06 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:23.694182 2982 raft_consensus.cc:3037] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:23.695276 2791 raft_consensus.cc:2454] T e62bf4b11eab47f193f28d71e8bd8bdf P dc0d993490e247b9a6aa70eedc63eb06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a7053094ff74e82b9a176d1db50adca in term 1.
I20260804 14:53:23.695279 2982 raft_consensus.cc:2454] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a7053094ff74e82b9a176d1db50adca in term 1.
I20260804 14:53:23.695624 2511 leader_election.cc:302] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a7053094ff74e82b9a176d1db50adca, dc0d993490e247b9a6aa70eedc63eb06; no voters:
I20260804 14:53:23.695860 3086 raft_consensus.cc:2781] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:23.695948 3086 raft_consensus.cc:686] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [term 1 LEADER]: Becoming Leader. State: Replica: 9a7053094ff74e82b9a176d1db50adca, State: Running, Role: LEADER
I20260804 14:53:23.696347 3086 consensus_queue.cc:237] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 } } peers { permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 } } peers { permanent_uuid: "9a7053094ff74e82b9a176d1db50adca" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40315 } }
W20260804 14:53:23.699394 2863 tablet.cc:2369] T e62bf4b11eab47f193f28d71e8bd8bdf P dc0d993490e247b9a6aa70eedc63eb06: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260804 14:53:23.699858 2410 catalog_manager.cc:5495] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a7053094ff74e82b9a176d1db50adca (127.24.6.129). New cstate: current_term: 1 leader_uuid: "9a7053094ff74e82b9a176d1db50adca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 } } peers { permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 } } peers { permanent_uuid: "9a7053094ff74e82b9a176d1db50adca" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40315 } } }
W20260804 14:53:23.709260 3051 tablet.cc:2369] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260804 14:53:23.718992 24602 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260804 14:53:23.720717 24602 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 9a7053094ff74e82b9a176d1db50adca to finish bootstrapping
I20260804 14:53:23.727862 24602 ts_itest-base.cc:233] Waiting for 1 tablets on tserver dc0d993490e247b9a6aa70eedc63eb06 to finish bootstrapping
I20260804 14:53:23.734227 24602 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 422632429a694bde88205930ec518cc6 to finish bootstrapping
I20260804 14:53:23.756297 24602 cluster_itest_util.cc:256] Not converged past 1 yet: 0.0 0.0 1.1
W20260804 14:53:23.766816 2670 tablet.cc:2369] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260804 14:53:23.859853 24602 cluster_itest_util.cc:256] Not converged past 1 yet: 0.0 0.0 1.1
I20260804 14:53:24.063246 24602 cluster_itest_util.cc:256] Not converged past 1 yet: 0.0 0.0 1.1
I20260804 14:53:24.129328 3086 consensus_queue.cc:1035] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [LEADER]: Connected to new peer: Peer: permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260804 14:53:24.135546 3085 consensus_queue.cc:1035] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260804 14:53:24.481065 24602 test_util.cc:268] Using random seed: -787130339
I20260804 14:53:24.789381 24602 test_util.cc:268] Using random seed: -786822026
I20260804 14:53:25.196283 24602 test_util.cc:268] Using random seed: -786415120
I20260804 14:53:25.635039 3105 raft_consensus.cc:482] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a7053094ff74e82b9a176d1db50adca)
I20260804 14:53:25.635284 3105 raft_consensus.cc:504] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 } } peers { permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 } } peers { permanent_uuid: "9a7053094ff74e82b9a176d1db50adca" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40315 } }
I20260804 14:53:25.636695 3105 leader_election.cc:288] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dc0d993490e247b9a6aa70eedc63eb06 (127.24.6.130:41505), 9a7053094ff74e82b9a176d1db50adca (127.24.6.129:40315)
I20260804 14:53:25.637430 3112 raft_consensus.cc:482] T e62bf4b11eab47f193f28d71e8bd8bdf P dc0d993490e247b9a6aa70eedc63eb06 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a7053094ff74e82b9a176d1db50adca)
I20260804 14:53:25.637615 3112 raft_consensus.cc:504] T e62bf4b11eab47f193f28d71e8bd8bdf P dc0d993490e247b9a6aa70eedc63eb06 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 } } peers { permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 } } peers { permanent_uuid: "9a7053094ff74e82b9a176d1db50adca" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40315 } }
I20260804 14:53:25.638849 3112 leader_election.cc:288] T e62bf4b11eab47f193f28d71e8bd8bdf P dc0d993490e247b9a6aa70eedc63eb06 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 422632429a694bde88205930ec518cc6 (127.24.6.131:41365), 9a7053094ff74e82b9a176d1db50adca (127.24.6.129:40315)
I20260804 14:53:25.641902 2791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e62bf4b11eab47f193f28d71e8bd8bdf" candidate_uuid: "422632429a694bde88205930ec518cc6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dc0d993490e247b9a6aa70eedc63eb06" is_pre_election: true
I20260804 14:53:25.642119 2791 raft_consensus.cc:2454] T e62bf4b11eab47f193f28d71e8bd8bdf P dc0d993490e247b9a6aa70eedc63eb06 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 422632429a694bde88205930ec518cc6 in term 1.
I20260804 14:53:25.642604 2890 leader_election.cc:302] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 422632429a694bde88205930ec518cc6, dc0d993490e247b9a6aa70eedc63eb06; no voters:
I20260804 14:53:25.643006 3105 raft_consensus.cc:2781] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260804 14:53:25.643105 3105 raft_consensus.cc:482] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9a7053094ff74e82b9a176d1db50adca)
I20260804 14:53:25.643157 3105 raft_consensus.cc:3037] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [term 1 FOLLOWER]: Advancing to term 2
I20260804 14:53:25.643615 2982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e62bf4b11eab47f193f28d71e8bd8bdf" candidate_uuid: "dc0d993490e247b9a6aa70eedc63eb06" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "422632429a694bde88205930ec518cc6" is_pre_election: true
I20260804 14:53:25.644450 3105 raft_consensus.cc:504] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 } } peers { permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 } } peers { permanent_uuid: "9a7053094ff74e82b9a176d1db50adca" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40315 } }
I20260804 14:53:25.644770 2982 raft_consensus.cc:2379] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dc0d993490e247b9a6aa70eedc63eb06 in current term 2: Already voted for candidate 422632429a694bde88205930ec518cc6 in this term.
I20260804 14:53:25.644945 3105 leader_election.cc:288] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [CANDIDATE]: Term 2 election: Requested vote from peers dc0d993490e247b9a6aa70eedc63eb06 (127.24.6.130:41505), 9a7053094ff74e82b9a176d1db50adca (127.24.6.129:40315)
I20260804 14:53:25.645447 2791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e62bf4b11eab47f193f28d71e8bd8bdf" candidate_uuid: "422632429a694bde88205930ec518cc6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dc0d993490e247b9a6aa70eedc63eb06"
I20260804 14:53:25.645642 2791 raft_consensus.cc:3037] T e62bf4b11eab47f193f28d71e8bd8bdf P dc0d993490e247b9a6aa70eedc63eb06 [term 1 FOLLOWER]: Advancing to term 2
I20260804 14:53:25.646628 2791 raft_consensus.cc:2454] T e62bf4b11eab47f193f28d71e8bd8bdf P dc0d993490e247b9a6aa70eedc63eb06 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 422632429a694bde88205930ec518cc6 in term 2.
I20260804 14:53:25.647091 2890 leader_election.cc:302] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 422632429a694bde88205930ec518cc6, dc0d993490e247b9a6aa70eedc63eb06; no voters:
I20260804 14:53:25.647425 3105 raft_consensus.cc:2781] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [term 2 FOLLOWER]: Leader election won for term 2
I20260804 14:53:25.647672 3105 raft_consensus.cc:686] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [term 2 LEADER]: Becoming Leader. State: Replica: 422632429a694bde88205930ec518cc6, State: Running, Role: LEADER
I20260804 14:53:25.648049 3105 consensus_queue.cc:237] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 } } peers { permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 } } peers { permanent_uuid: "9a7053094ff74e82b9a176d1db50adca" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40315 } }
I20260804 14:53:25.650822 2410 catalog_manager.cc:5495] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 reported cstate change: term changed from 1 to 2, leader changed from 9a7053094ff74e82b9a176d1db50adca (127.24.6.129) to 422632429a694bde88205930ec518cc6 (127.24.6.131). New cstate: current_term: 2 leader_uuid: "422632429a694bde88205930ec518cc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 } } peers { permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 } } peers { permanent_uuid: "9a7053094ff74e82b9a176d1db50adca" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40315 } } }
I20260804 14:53:25.703665 24602 test_util.cc:268] Using random seed: -785907740
I20260804 14:53:25.706955 24602 test_util.cc:268] Using random seed: -785904448
I20260804 14:53:25.725514 24602 test_workload.cc:401] TestWorkload: Skipping table creation because table TestTable already exists
I20260804 14:53:25.725975 24602 raft_consensus-itest-base.cc:268] Waiting until we've written at least 4MB...
I20260804 14:53:25.743803 2791 raft_consensus.cc:1261] T e62bf4b11eab47f193f28d71e8bd8bdf P dc0d993490e247b9a6aa70eedc63eb06 [term 2 FOLLOWER]: Refusing update from remote peer 422632429a694bde88205930ec518cc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260804 14:53:25.744606 3105 consensus_queue.cc:1035] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260804 14:53:25.809204 2863 maintenance_manager.cc:419] P dc0d993490e247b9a6aa70eedc63eb06: Scheduling FlushMRSOp(e62bf4b11eab47f193f28d71e8bd8bdf): 2366903 bytes log retention, and flush 4038921 bytes memory
I20260804 14:53:25.818550 3051 maintenance_manager.cc:419] P 422632429a694bde88205930ec518cc6: Scheduling FlushMRSOp(e62bf4b11eab47f193f28d71e8bd8bdf): 2761362 bytes log retention, and flush 4038921 bytes memory
I20260804 14:53:25.866724 24602 raft_consensus-itest-base.cc:274] Waiting for log GC on 422632429a694bde88205930ec518cc6
I20260804 14:53:25.895143 2701 maintenance_manager.cc:643] P dc0d993490e247b9a6aa70eedc63eb06: FlushMRSOp(e62bf4b11eab47f193f28d71e8bd8bdf) complete. Timing: real 0.085s user 0.053s sys 0.004s Metrics: {"bytes_written":135869,"cfile_init":1,"compiler_manager_pool.queue_time_us":313,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":958,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":22,"thread_start_us":374,"threads_started":2}
I20260804 14:53:25.896638 2863 maintenance_manager.cc:419] P dc0d993490e247b9a6aa70eedc63eb06: Scheduling LogGCOp(e62bf4b11eab47f193f28d71e8bd8bdf): free 2366903 bytes of WAL
I20260804 14:53:25.897361 2701 log_reader.cc:386] T e62bf4b11eab47f193f28d71e8bd8bdf: removed 3 log segments from log reader
I20260804 14:53:25.897526 2701 log.cc:1076] T e62bf4b11eab47f193f28d71e8bd8bdf P dc0d993490e247b9a6aa70eedc63eb06: Deleting log segment in path: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/wals/e62bf4b11eab47f193f28d71e8bd8bdf/wal-000000001 (ops 1-7)
I20260804 14:53:25.897732 2701 log.cc:1076] T e62bf4b11eab47f193f28d71e8bd8bdf P dc0d993490e247b9a6aa70eedc63eb06: Deleting log segment in path: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/wals/e62bf4b11eab47f193f28d71e8bd8bdf/wal-000000002 (ops 8-14)
I20260804 14:53:25.898017 2701 log.cc:1076] T e62bf4b11eab47f193f28d71e8bd8bdf P dc0d993490e247b9a6aa70eedc63eb06: Deleting log segment in path: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/wals/e62bf4b11eab47f193f28d71e8bd8bdf/wal-000000003 (ops 15-20)
I20260804 14:53:25.899111 2701 maintenance_manager.cc:643] P dc0d993490e247b9a6aa70eedc63eb06: LogGCOp(e62bf4b11eab47f193f28d71e8bd8bdf) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {}
I20260804 14:53:25.899562 2863 maintenance_manager.cc:419] P dc0d993490e247b9a6aa70eedc63eb06: Scheduling FlushMRSOp(e62bf4b11eab47f193f28d71e8bd8bdf): 1709323 bytes log retention, and flush 2998537 bytes memory
I20260804 14:53:25.904992 2892 maintenance_manager.cc:643] P 422632429a694bde88205930ec518cc6: FlushMRSOp(e62bf4b11eab47f193f28d71e8bd8bdf) complete. Timing: real 0.086s user 0.054s sys 0.000s Metrics: {"bytes_written":135937,"cfile_init":1,"compiler_manager_pool.queue_time_us":345,"dirs.queue_time_us":251,"dirs.run_cpu_time_us":166,"dirs.run_wall_time_us":765,"drs_written":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4,"lbm_write_time_us":835,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":25,"thread_start_us":342,"threads_started":2}
I20260804 14:53:25.906296 3051 maintenance_manager.cc:419] P 422632429a694bde88205930ec518cc6: Scheduling LogGCOp(e62bf4b11eab47f193f28d71e8bd8bdf): free 2761362 bytes of WAL
I20260804 14:53:25.906711 2892 log_reader.cc:386] T e62bf4b11eab47f193f28d71e8bd8bdf: removed 3 log segments from log reader
I20260804 14:53:25.906822 2892 log.cc:1076] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6: Deleting log segment in path: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal/wals/e62bf4b11eab47f193f28d71e8bd8bdf/wal-000000001 (ops 1-9)
I20260804 14:53:25.906960 2892 log.cc:1076] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6: Deleting log segment in path: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal/wals/e62bf4b11eab47f193f28d71e8bd8bdf/wal-000000002 (ops 10-16)
I20260804 14:53:25.907032 2892 log.cc:1076] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6: Deleting log segment in path: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal/wals/e62bf4b11eab47f193f28d71e8bd8bdf/wal-000000003 (ops 17-23)
I20260804 14:53:25.908048 24602 raft_consensus-itest-base.cc:279] Log GC complete on 422632429a694bde88205930ec518cc6
I20260804 14:53:25.908656 2892 maintenance_manager.cc:643] P 422632429a694bde88205930ec518cc6: LogGCOp(e62bf4b11eab47f193f28d71e8bd8bdf) complete. Timing: real 0.002s user 0.000s sys 0.001s Metrics: {}
I20260804 14:53:25.909050 3051 maintenance_manager.cc:419] P 422632429a694bde88205930ec518cc6: Scheduling FlushMRSOp(e62bf4b11eab47f193f28d71e8bd8bdf): 1840988 bytes log retention, and flush 1958153 bytes memory
I20260804 14:53:25.926285 2701 maintenance_manager.cc:643] P dc0d993490e247b9a6aa70eedc63eb06: FlushMRSOp(e62bf4b11eab47f193f28d71e8bd8bdf) complete. Timing: real 0.027s user 0.021s sys 0.003s Metrics: {"bytes_written":135803,"cfile_init":1,"dirs.queue_time_us":62,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":1941,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":715,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":15}
I20260804 14:53:25.927675 2863 maintenance_manager.cc:419] P dc0d993490e247b9a6aa70eedc63eb06: Scheduling CompactRowSetsOp(e62bf4b11eab47f193f28d71e8bd8bdf): perf score=0.982854
I20260804 14:53:25.932885 2892 maintenance_manager.cc:643] P 422632429a694bde88205930ec518cc6: FlushMRSOp(e62bf4b11eab47f193f28d71e8bd8bdf) complete. Timing: real 0.024s user 0.015s sys 0.005s Metrics: {"bytes_written":135803,"cfile_init":1,"dirs.queue_time_us":78,"dirs.run_cpu_time_us":214,"dirs.run_wall_time_us":1527,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":706,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":12}
I20260804 14:53:25.934209 3051 maintenance_manager.cc:419] P 422632429a694bde88205930ec518cc6: Scheduling CompactRowSetsOp(e62bf4b11eab47f193f28d71e8bd8bdf): perf score=0.982848
I20260804 14:53:25.991884 2701 maintenance_manager.cc:643] P dc0d993490e247b9a6aa70eedc63eb06: CompactRowSetsOp(e62bf4b11eab47f193f28d71e8bd8bdf) complete. Timing: real 0.064s user 0.059s sys 0.004s Metrics: {"bytes_written":136001,"cfile_cache_hit":10,"cfile_cache_hit_bytes":262392,"cfile_cache_miss":26,"cfile_cache_miss_bytes":264308,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":116,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":2022,"drs_written":1,"lbm_read_time_us":916,"lbm_reads_lt_1ms":54,"lbm_write_time_us":847,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":284,"rows_written":37,"thread_start_us":443,"threads_started":5}
I20260804 14:53:25.993131 2863 maintenance_manager.cc:419] P dc0d993490e247b9a6aa70eedc63eb06: Scheduling UndoDeltaBlockGCOp(e62bf4b11eab47f193f28d71e8bd8bdf): 551 bytes on disk
I20260804 14:53:25.994127 2701 maintenance_manager.cc:643] P dc0d993490e247b9a6aa70eedc63eb06: UndoDeltaBlockGCOp(e62bf4b11eab47f193f28d71e8bd8bdf) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20260804 14:53:25.996845 2892 maintenance_manager.cc:643] P 422632429a694bde88205930ec518cc6: CompactRowSetsOp(e62bf4b11eab47f193f28d71e8bd8bdf) complete. Timing: real 0.062s user 0.062s sys 0.000s Metrics: {"bytes_written":136001,"cfile_cache_hit":10,"cfile_cache_hit_bytes":262394,"cfile_cache_miss":26,"cfile_cache_miss_bytes":264452,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":1142,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":1212,"drs_written":1,"lbm_read_time_us":838,"lbm_reads_lt_1ms":54,"lbm_write_time_us":799,"lbm_writes_lt_1ms":23,"mutex_wait_us":22,"num_input_rowsets":2,"peak_mem_usage":324,"rows_written":37,"spinlock_wait_cycles":3328,"thread_start_us":431,"threads_started":5}
I20260804 14:53:25.997666 3051 maintenance_manager.cc:419] P 422632429a694bde88205930ec518cc6: Scheduling UndoDeltaBlockGCOp(e62bf4b11eab47f193f28d71e8bd8bdf): 551 bytes on disk
I20260804 14:53:25.998272 2892 maintenance_manager.cc:643] P 422632429a694bde88205930ec518cc6: UndoDeltaBlockGCOp(e62bf4b11eab47f193f28d71e8bd8bdf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
W20260804 14:53:27.204111 2696 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.24.6.129:40315 for RPC RequestConsensusVote timed out after 1.563s (ON_OUTBOUND_QUEUE)
W20260804 14:53:27.390563 2888 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.24.6.129:40315 for RPC RequestConsensusVote timed out after 1.752s (ON_OUTBOUND_QUEUE)
I20260804 14:53:27.909850 24602 raft_consensus-itest-base.cc:295] Servers converged with original term 2
I20260804 14:53:27.909956 24602 raft_consensus-itest-base.cc:300] Resuming 9a7053094ff74e82b9a176d1db50adca
I20260804 14:53:27.915707 2791 raft_consensus.cc:1226] T e62bf4b11eab47f193f28d71e8bd8bdf P dc0d993490e247b9a6aa70eedc63eb06 [term 2 FOLLOWER]: Rejecting Update request from peer 9a7053094ff74e82b9a176d1db50adca for earlier term 1. Current term is 2. Ops: []
I20260804 14:53:27.915926 2982 raft_consensus.cc:1226] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [term 2 LEADER]: Rejecting Update request from peer 9a7053094ff74e82b9a176d1db50adca for earlier term 1. Current term is 2. Ops: []
I20260804 14:53:27.916476 3085 consensus_queue.cc:1046] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 39, Time since last communication: 0.000s
I20260804 14:53:27.916700 3085 consensus_queue.cc:1225] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [LEADER]: Peer 422632429a694bde88205930ec518cc6 log is divergent from this leader: its last log entry 2.39 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 39
I20260804 14:53:27.916781 3085 raft_consensus.cc:3032] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [term 1 LEADER]: Stepping down as leader of term 1
I20260804 14:53:27.916821 3085 raft_consensus.cc:729] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9a7053094ff74e82b9a176d1db50adca, State: Running, Role: LEADER
I20260804 14:53:27.917016 3085 consensus_queue.cc:260] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 } } peers { permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 } } peers { permanent_uuid: "9a7053094ff74e82b9a176d1db50adca" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40315 } }
I20260804 14:53:27.917759 3086 consensus_queue.cc:1046] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 39, Time since last communication: 0.000s
I20260804 14:53:27.918582 3085 raft_consensus.cc:3037] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [term 1 FOLLOWER]: Advancing to term 2
I20260804 14:53:27.919728 2602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e62bf4b11eab47f193f28d71e8bd8bdf" candidate_uuid: "dc0d993490e247b9a6aa70eedc63eb06" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a7053094ff74e82b9a176d1db50adca" is_pre_election: true
I20260804 14:53:27.919879 2594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e62bf4b11eab47f193f28d71e8bd8bdf" candidate_uuid: "422632429a694bde88205930ec518cc6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a7053094ff74e82b9a176d1db50adca"
I20260804 14:53:27.920241 2601 raft_consensus.cc:1261] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [term 2 FOLLOWER]: Refusing update from remote peer 422632429a694bde88205930ec518cc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260804 14:53:27.920743 2603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e62bf4b11eab47f193f28d71e8bd8bdf" candidate_uuid: "422632429a694bde88205930ec518cc6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a7053094ff74e82b9a176d1db50adca" is_pre_election: true
I20260804 14:53:27.920926 2603 raft_consensus.cc:2454] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 422632429a694bde88205930ec518cc6 in term 2.
I20260804 14:53:27.921329 2696 leader_election.cc:302] T e62bf4b11eab47f193f28d71e8bd8bdf P dc0d993490e247b9a6aa70eedc63eb06 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc0d993490e247b9a6aa70eedc63eb06; no voters: 422632429a694bde88205930ec518cc6, 9a7053094ff74e82b9a176d1db50adca
I20260804 14:53:27.921969 3194 consensus_queue.cc:1035] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a7053094ff74e82b9a176d1db50adca" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260804 14:53:27.922055 3201 raft_consensus.cc:2726] T e62bf4b11eab47f193f28d71e8bd8bdf P dc0d993490e247b9a6aa70eedc63eb06 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260804 14:53:27.922428 3194 consensus_queue.cc:740] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [LEADER]: The logs necessary to catch up peer 9a7053094ff74e82b9a176d1db50adca have been garbage collected. The follower will never be able to catch up (Not found: failed to read ops 2..32: Segment 1 which contained index 2 has been GCed)
I20260804 14:53:27.922830 3194 raft_consensus.cc:998] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6: Attempting to remove follower 9a7053094ff74e82b9a176d1db50adca from the Raft config. Reason: The logs necessary to catch up peer 9a7053094ff74e82b9a176d1db50adca have been garbage collected. The replica will never be able to catch up
I20260804 14:53:27.923732 3194 consensus_queue.cc:237] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 2.39, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 } } peers { permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 } }
I20260804 14:53:27.925434 2791 raft_consensus.cc:1261] T e62bf4b11eab47f193f28d71e8bd8bdf P dc0d993490e247b9a6aa70eedc63eb06 [term 2 FOLLOWER]: Refusing update from remote peer 422632429a694bde88205930ec518cc6: Log matching property violated. Preceding OpId in replica: term: 2 index: 39. Preceding OpId from leader: term: 2 index: 40. (index mismatch)
I20260804 14:53:27.925916 3203 consensus_queue.cc:1035] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20260804 14:53:27.928620 3194 raft_consensus.cc:2932] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [term 2 LEADER]: Committing config change with OpId 2.40: config changed from index -1 to 40, VOTER 9a7053094ff74e82b9a176d1db50adca (127.24.6.129) evicted. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 } } peers { permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 } } }
I20260804 14:53:27.929193 2791 raft_consensus.cc:2932] T e62bf4b11eab47f193f28d71e8bd8bdf P dc0d993490e247b9a6aa70eedc63eb06 [term 2 FOLLOWER]: Committing config change with OpId 2.40: config changed from index -1 to 40, VOTER 9a7053094ff74e82b9a176d1db50adca (127.24.6.129) evicted. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 } } peers { permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 } } }
I20260804 14:53:27.931869 2410 catalog_manager.cc:5495] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 reported cstate change: config changed from index -1 to 40, VOTER 9a7053094ff74e82b9a176d1db50adca (127.24.6.129) evicted. New cstate: current_term: 2 leader_uuid: "422632429a694bde88205930ec518cc6" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 } } peers { permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 } } }
I20260804 14:53:27.935678 2574 tablet_service.cc:1514] Processing DeleteTablet for tablet e62bf4b11eab47f193f28d71e8bd8bdf with delete_type TABLET_DATA_TOMBSTONED (TS 9a7053094ff74e82b9a176d1db50adca not found in new config with opid_index 40) from {username='slave'} at 127.0.0.1:35586
I20260804 14:53:27.936236 3214 tablet_replica.cc:330] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca: stopping tablet replica
I20260804 14:53:27.936631 3214 raft_consensus.cc:2229] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [term 2 FOLLOWER]: Raft consensus shutting down.
I20260804 14:53:27.936892 3214 raft_consensus.cc:2258] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [term 2 FOLLOWER]: Raft consensus is shut down!
I20260804 14:53:27.937979 3214 ts_tablet_manager.cc:1905] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260804 14:53:27.940481 3214 ts_tablet_manager.cc:1918] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260804 14:53:27.940634 3214 log.cc:1196] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca: Deleting WAL directory at /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/wals/e62bf4b11eab47f193f28d71e8bd8bdf
I20260804 14:53:27.941468 2387 catalog_manager.cc:4841] TS 9a7053094ff74e82b9a176d1db50adca (127.24.6.129:40315): tablet e62bf4b11eab47f193f28d71e8bd8bdf (table TestTable [id=375975d53e964d7ea790eb3690c3fd2c]) successfully deleted
I20260804 14:53:27.946463 2660 generic_service.cc:175] {username='slave'} at 127.0.0.1:35588 changed flags via RPC: tablet_copy_fault_crash_on_fetch_all from '0' to '1.0'
I20260804 14:53:27.948252 2982 consensus_queue.cc:237] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 2.40, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 } } peers { permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 } } peers { permanent_uuid: "9a7053094ff74e82b9a176d1db50adca" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40315 } attrs { } }
I20260804 14:53:27.949836 2791 raft_consensus.cc:1261] T e62bf4b11eab47f193f28d71e8bd8bdf P dc0d993490e247b9a6aa70eedc63eb06 [term 2 FOLLOWER]: Refusing update from remote peer 422632429a694bde88205930ec518cc6: Log matching property violated. Preceding OpId in replica: term: 2 index: 40. Preceding OpId from leader: term: 2 index: 41. (index mismatch)
W20260804 14:53:27.950286 2888 consensus_peers.cc:487] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 -> Peer 9a7053094ff74e82b9a176d1db50adca (127.24.6.129:40315): Couldn't send request to peer 9a7053094ff74e82b9a176d1db50adca. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260804 14:53:27.950314 3203 consensus_queue.cc:1035] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20260804 14:53:27.952435 3194 raft_consensus.cc:2932] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [term 2 LEADER]: Committing config change with OpId 2.41: config changed from index 40 to 41, VOTER 9a7053094ff74e82b9a176d1db50adca (127.24.6.129) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 } } peers { permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 } } peers { permanent_uuid: "9a7053094ff74e82b9a176d1db50adca" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40315 } attrs { } } }
I20260804 14:53:27.952952 2791 raft_consensus.cc:2932] T e62bf4b11eab47f193f28d71e8bd8bdf P dc0d993490e247b9a6aa70eedc63eb06 [term 2 FOLLOWER]: Committing config change with OpId 2.41: config changed from index 40 to 41, VOTER 9a7053094ff74e82b9a176d1db50adca (127.24.6.129) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 } } peers { permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 } } peers { permanent_uuid: "9a7053094ff74e82b9a176d1db50adca" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40315 } attrs { } } }
I20260804 14:53:27.955423 2407 catalog_manager.cc:5495] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 reported cstate change: config changed from index 40 to 41, VOTER 9a7053094ff74e82b9a176d1db50adca (127.24.6.129) added. New cstate: current_term: 2 leader_uuid: "422632429a694bde88205930ec518cc6" committed_config { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 } } peers { permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 } } peers { permanent_uuid: "9a7053094ff74e82b9a176d1db50adca" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40315 } attrs { } } }
I20260804 14:53:28.412334 3216 ts_tablet_manager.cc:927] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca: Initiating tablet copy from peer 422632429a694bde88205930ec518cc6 (127.24.6.131:41365)
I20260804 14:53:28.413103 3216 tablet_copy_client.cc:316] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca: tablet copy: Beginning tablet copy session from remote peer at address 127.24.6.131:41365
I20260804 14:53:28.414180 3007 tablet_copy_service.cc:140] P 422632429a694bde88205930ec518cc6: Received BeginTabletCopySession request for tablet e62bf4b11eab47f193f28d71e8bd8bdf from peer 9a7053094ff74e82b9a176d1db50adca ({username='slave'} at 127.24.6.129:45827)
I20260804 14:53:28.414420 3007 tablet_copy_service.cc:161] P 422632429a694bde88205930ec518cc6: Beginning new tablet copy session on tablet e62bf4b11eab47f193f28d71e8bd8bdf from peer 9a7053094ff74e82b9a176d1db50adca at {username='slave'} at 127.24.6.129:45827: session id = 9a7053094ff74e82b9a176d1db50adca-e62bf4b11eab47f193f28d71e8bd8bdf
I20260804 14:53:28.416373 3007 tablet_copy_source_session.cc:205] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6: Tablet Copy: opened 5 blocks and 3 log segments
I20260804 14:53:28.418620 3216 ts_tablet_manager.cc:1905] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260804 14:53:28.421258 3216 ts_tablet_manager.cc:1918] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1
I20260804 14:53:28.421504 3216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e62bf4b11eab47f193f28d71e8bd8bdf. 1 dirs total, 0 dirs full, 0 dirs failed
W20260804 14:53:28.422928 3216 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260804 14:53:28.422991 3216 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260804 14:53:28.423028 3216 fault_injection.cc:59] injecting fault for kudu (pid 2491): FLAGS_tablet_copy_fault_crash_on_fetch_all (process will exit)
I20260804 14:53:28.506464 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 2674
I20260804 14:53:28.521122 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 2868
I20260804 14:53:28.545814 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 2370
I20260804 14:53:28.562649 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.6.190:33693
--webserver_interface=127.24.6.190
--webserver_port=42645
--builtin_ntp_servers=127.24.6.148:36237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.6.190:33693
--master_add_server_when_underreplicated=false
--raft_prepare_replacement_before_eviction=false with env {}
W20260804 14:53:28.704834 3218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:28.705261 3218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:28.705302 3218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:28.709971 3218 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260804 14:53:28.710079 3218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260804 14:53:28.710098 3218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:28.710116 3218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260804 14:53:28.710134 3218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260804 14:53:28.715453 3218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:36237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
--master_add_server_when_underreplicated=false
--ipki_ca_key_size=768
--master_addresses=127.24.6.190:33693
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.6.190:33693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.24.6.190
--webserver_port=42645
--never_fsync=true
--heap_profile_path=/tmp/kudu.3218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:28.717164 3218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:28.718962 3218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:28.729856 3227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:28.729853 3228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:28.729853 3230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:28.730324 3218 server_base.cc:1034] running on GCE node
I20260804 14:53:28.731184 3218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:28.732753 3218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:28.734012 3218 hybrid_clock.cc:648] HybridClock initialized: now 1785855208733988 us; error 47 us; skew 500 ppm
I20260804 14:53:28.736974 3218 webserver.cc:460] Webserver started at http://127.24.6.190:42645/ using document root <none> and password file <none>
I20260804 14:53:28.737906 3218 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:28.738092 3218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:28.743422 3218 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260804 14:53:28.746373 3238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:28.748008 3218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260804 14:53:28.748204 3218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2f5202c683d8428da3a7273ca55e7827"
format_stamp: "Formatted at 2026-08-04 14:53:22 on dist-test-slave-92m1"
I20260804 14:53:28.748878 3218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:28.765400 3218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:28.766381 3218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:28.766664 3218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:28.775794 3218 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.190:33693
I20260804 14:53:28.775835 3313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.190:33693 every 8 connection(s)
I20260804 14:53:28.777092 3218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260804 14:53:28.782862 3314 sys_catalog.cc:263] Verifying existing consensus state
I20260804 14:53:28.784677 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 3218
I20260804 14:53:28.785212 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.131:41365
--local_ip_for_outbound_sockets=127.24.6.131
--tserver_master_addrs=127.24.6.190:33693
--webserver_port=45865
--webserver_interface=127.24.6.131
--builtin_ntp_servers=127.24.6.148:36237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20260804 14:53:28.785214 3314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827: Bootstrap starting.
I20260804 14:53:28.803052 3314 log.cc:824] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:28.815747 3314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=5 ignored=0} mutations{seen=5 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260804 14:53:28.816278 3314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827: Bootstrap complete.
I20260804 14:53:28.821344 3314 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5202c683d8428da3a7273ca55e7827" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 33693 } }
I20260804 14:53:28.821879 3314 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f5202c683d8428da3a7273ca55e7827, State: Initialized, Role: FOLLOWER
I20260804 14:53:28.822484 3314 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5202c683d8428da3a7273ca55e7827" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 33693 } }
I20260804 14:53:28.822659 3314 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260804 14:53:28.822765 3314 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260804 14:53:28.822907 3314 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [term 1 FOLLOWER]: Advancing to term 2
I20260804 14:53:28.825191 3314 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5202c683d8428da3a7273ca55e7827" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 33693 } }
I20260804 14:53:28.825623 3314 leader_election.cc:302] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f5202c683d8428da3a7273ca55e7827; no voters:
I20260804 14:53:28.826054 3314 leader_election.cc:288] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260804 14:53:28.826169 3319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [term 2 FOLLOWER]: Leader election won for term 2
I20260804 14:53:28.826421 3319 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [term 2 LEADER]: Becoming Leader. State: Replica: 2f5202c683d8428da3a7273ca55e7827, State: Running, Role: LEADER
I20260804 14:53:28.826896 3319 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5202c683d8428da3a7273ca55e7827" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 33693 } }
I20260804 14:53:28.827811 3314 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [sys.catalog]: configured and running, proceeding with master startup.
I20260804 14:53:28.829197 3320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2f5202c683d8428da3a7273ca55e7827" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5202c683d8428da3a7273ca55e7827" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 33693 } } }
I20260804 14:53:28.829346 3320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [sys.catalog]: This master's current role is: LEADER
I20260804 14:53:28.829846 3321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f5202c683d8428da3a7273ca55e7827. Latest consensus state: current_term: 2 leader_uuid: "2f5202c683d8428da3a7273ca55e7827" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5202c683d8428da3a7273ca55e7827" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 33693 } } }
I20260804 14:53:28.830015 3321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827 [sys.catalog]: This master's current role is: LEADER
I20260804 14:53:28.830368 3332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260804 14:53:28.835836 3332 catalog_manager.cc:669] Loaded metadata for table TestTable [id=375975d53e964d7ea790eb3690c3fd2c]
I20260804 14:53:28.837772 3332 tablet_loader.cc:96] loaded metadata for tablet e62bf4b11eab47f193f28d71e8bd8bdf (table TestTable [id=375975d53e964d7ea790eb3690c3fd2c])
I20260804 14:53:28.838049 3332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260804 14:53:28.839092 3332 catalog_manager.cc:1259] Loaded cluster ID: 68b0e8e5dd2b4878a25d5705a60e9fd0
I20260804 14:53:28.839185 3332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260804 14:53:28.842315 3332 catalog_manager.cc:1504] Loading token signing keys...
I20260804 14:53:28.843372 3332 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2f5202c683d8428da3a7273ca55e7827: Loaded TSK: 0
I20260804 14:53:28.844901 3332 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260804 14:53:28.921059 3317 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20260804 14:53:28.921366 3317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:28.921388 3317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:28.921474 3317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:28.925693 3317 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20260804 14:53:28.925755 3317 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20260804 14:53:28.925778 3317 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260804 14:53:28.925801 3317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:28.925892 3317 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20260804 14:53:28.925933 3317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.131
I20260804 14:53:28.931550 3317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:36237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.131:41365
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.24.6.131
--webserver_port=45865
--tserver_master_addrs=127.24.6.190:33693
--never_fsync=true
--heap_profile_path=/tmp/kudu.3317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.131
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:28.932732 3317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:28.934207 3317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:28.944348 3350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:28.944496 3349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:28.944342 3352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:28.945226 3317 server_base.cc:1034] running on GCE node
I20260804 14:53:28.945701 3317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:28.946494 3317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:28.947674 3317 hybrid_clock.cc:648] HybridClock initialized: now 1785855208947657 us; error 38 us; skew 500 ppm
I20260804 14:53:28.949862 3317 webserver.cc:460] Webserver started at http://127.24.6.131:45865/ using document root <none> and password file <none>
I20260804 14:53:28.950512 3317 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:28.950599 3317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:28.955165 3317 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260804 14:53:28.961612 3373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:28.962937 3317 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.002s
I20260804 14:53:28.963096 3317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "422632429a694bde88205930ec518cc6"
format_stamp: "Formatted at 2026-08-04 14:53:23 on dist-test-slave-92m1"
I20260804 14:53:28.963763 3317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 5
Total live bytes: 136552
Total live bytes (after alignment): 155648
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:28.989316 3317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:28.990159 3317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:28.990404 3317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:28.991091 3317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:53:28.992539 3382 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260804 14:53:28.997785 3317 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260804 14:53:28.997918 3317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260804 14:53:28.998013 3317 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260804 14:53:28.999927 3317 ts_tablet_manager.cc:610] Registered 1 tablets
I20260804 14:53:29.000039 3317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260804 14:53:29.000351 3382 tablet_bootstrap.cc:492] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6: Bootstrap starting.
I20260804 14:53:29.011497 3317 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.131:41365
I20260804 14:53:29.011581 3539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.131:41365 every 8 connection(s)
I20260804 14:53:29.012627 3317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260804 14:53:29.014820 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 3317
I20260804 14:53:29.015455 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.129:40315
--local_ip_for_outbound_sockets=127.24.6.129
--tserver_master_addrs=127.24.6.190:33693
--webserver_port=43463
--webserver_interface=127.24.6.129
--builtin_ntp_servers=127.24.6.148:36237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20260804 14:53:29.015498 3382 log.cc:824] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:29.027552 3382 tablet_bootstrap.cc:492] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6: Bootstrap replayed 1/3 log segments. Stats: ops{read=7 overwritten=0 applied=5 ignored=5} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=3. Pending: 2 replicates
I20260804 14:53:29.028671 3540 heartbeater.cc:347] Connected to a master server at 127.24.6.190:33693
I20260804 14:53:29.029002 3540 heartbeater.cc:464] Registering TS with master...
I20260804 14:53:29.029551 3540 heartbeater.cc:511] Master 127.24.6.190:33693 requested a full tablet report, sending...
I20260804 14:53:29.032644 3265 ts_manager.cc:194] Registered new tserver with Master: 422632429a694bde88205930ec518cc6 (127.24.6.131:41365)
I20260804 14:53:29.034868 3382 tablet_bootstrap.cc:492] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6: Bootstrap replayed 2/3 log segments. Stats: ops{read=14 overwritten=0 applied=13 ignored=13} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=3. Pending: 1 replicates
I20260804 14:53:29.034934 3265 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.131:52063
I20260804 14:53:29.037964 3382 tablet_bootstrap.cc:492] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6: Bootstrap replayed 3/3 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=16} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=3. Pending: 0 replicates
I20260804 14:53:29.038551 3382 tablet_bootstrap.cc:492] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6: Bootstrap complete.
I20260804 14:53:29.039753 3382 ts_tablet_manager.cc:1397] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6: Time spent bootstrapping tablet: real 0.040s user 0.030s sys 0.008s
I20260804 14:53:29.043159 3382 raft_consensus.cc:348] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 } } peers { permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 } } peers { permanent_uuid: "9a7053094ff74e82b9a176d1db50adca" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40315 } attrs { } }
I20260804 14:53:29.043881 3382 raft_consensus.cc:729] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 422632429a694bde88205930ec518cc6, State: Initialized, Role: FOLLOWER
I20260804 14:53:29.044472 3382 consensus_queue.cc:260] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 41, Last appended: 2.41, Last appended by leader: 41, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 } } peers { permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 } } peers { permanent_uuid: "9a7053094ff74e82b9a176d1db50adca" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40315 } attrs { } }
I20260804 14:53:29.045324 3540 heartbeater.cc:503] Master 127.24.6.190:33693 was elected leader, sending a full tablet report...
I20260804 14:53:29.045713 3382 ts_tablet_manager.cc:1428] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260804 14:53:29.114825 3543 maintenance_manager.cc:419] P 422632429a694bde88205930ec518cc6: Scheduling UndoDeltaBlockGCOp(e62bf4b11eab47f193f28d71e8bd8bdf): 551 bytes on disk
I20260804 14:53:29.116024 3380 maintenance_manager.cc:643] P 422632429a694bde88205930ec518cc6: UndoDeltaBlockGCOp(e62bf4b11eab47f193f28d71e8bd8bdf) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
W20260804 14:53:29.156002 3548 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20260804 14:53:29.156447 3548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:29.156513 3548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:29.156589 3548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:29.161484 3548 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20260804 14:53:29.161609 3548 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20260804 14:53:29.161661 3548 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260804 14:53:29.161741 3548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:29.161867 3548 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20260804 14:53:29.161926 3548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.129
I20260804 14:53:29.168259 3548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:36237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.129:40315
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.24.6.129
--webserver_port=43463
--tserver_master_addrs=127.24.6.190:33693
--never_fsync=true
--heap_profile_path=/tmp/kudu.3548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.129
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:29.169691 3548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:29.171478 3548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:29.183109 3558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:29.183104 3560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:29.183099 3556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:29.183389 3548 server_base.cc:1034] running on GCE node
I20260804 14:53:29.184044 3548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:29.184752 3548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:29.185956 3548 hybrid_clock.cc:648] HybridClock initialized: now 1785855209185914 us; error 53 us; skew 500 ppm
I20260804 14:53:29.188292 3548 webserver.cc:460] Webserver started at http://127.24.6.129:43463/ using document root <none> and password file <none>
I20260804 14:53:29.188958 3548 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:29.189073 3548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:29.193812 3548 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260804 14:53:29.196812 3568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:29.198446 3548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260804 14:53:29.198652 3548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "9a7053094ff74e82b9a176d1db50adca"
format_stamp: "Formatted at 2026-08-04 14:53:23 on dist-test-slave-92m1"
I20260804 14:53:29.199466 3548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:29.232161 3548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:29.233059 3548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:29.233331 3548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:29.234293 3548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:53:29.236006 3578 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260804 14:53:29.240557 3578 ts_tablet_manager.cc:1852] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_COPYING
I20260804 14:53:29.242154 3578 ts_tablet_manager.cc:1905] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260804 14:53:29.247277 3578 ts_tablet_manager.cc:1918] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260804 14:53:29.248387 3548 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260804 14:53:29.248485 3548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.000s sys 0.001s
I20260804 14:53:29.248610 3548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:53:29.248651 3548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:29.259538 3548 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.129:40315
I20260804 14:53:29.259584 3748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.129:40315 every 8 connection(s)
I20260804 14:53:29.260624 3548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260804 14:53:29.267910 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 3548
I20260804 14:53:29.268183 24602 test_util.cc:268] Using random seed: -782343209
I20260804 14:53:29.273298 3749 heartbeater.cc:347] Connected to a master server at 127.24.6.190:33693
I20260804 14:53:29.273667 3749 heartbeater.cc:464] Registering TS with master...
I20260804 14:53:29.274283 3749 heartbeater.cc:511] Master 127.24.6.190:33693 requested a full tablet report, sending...
I20260804 14:53:29.276078 3265 ts_manager.cc:194] Registered new tserver with Master: 9a7053094ff74e82b9a176d1db50adca (127.24.6.129:40315)
I20260804 14:53:29.276932 3265 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.129:35837
I20260804 14:53:29.282215 24602 test_workload.cc:401] TestWorkload: Skipping table creation because table TestTable already exists
W20260804 14:53:29.303354 3411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.303355 3413 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.303402 3408 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.303354 3409 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.308660 3756 meta_cache.cc:302] tablet e62bf4b11eab47f193f28d71e8bd8bdf: replica dc0d993490e247b9a6aa70eedc63eb06 (127.24.6.130:41505) has failed: Network error: Client connection negotiation failed: client connection to 127.24.6.130:41505: connect: Connection refused (error 111)
W20260804 14:53:29.326479 3413 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.327929 3413 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.328243 3409 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.338410 3409 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.347018 3409 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.347074 3413 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.358024 3413 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.358301 3409 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.367715 3409 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.368700 3409 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.380338 3409 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.380388 3413 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.381345 3413 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.397219 3413 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.421402 3413 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.438172 3413 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.438172 3411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.438218 3409 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.458675 3409 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.458740 3413 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.458976 3411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.491963 3411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.510826 3411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.533483 3411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.556980 3411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.558319 3411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.578999 3411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.603438 3411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.603452 3413 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.629513 3413 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.630430 3413 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.658113 3413 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.659186 3413 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.686508 3413 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.710335 3413 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.740226 3413 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.761600 3413 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.795876 3413 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.806154 3413 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.841859 3413 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.868655 3413 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.902302 3413 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.902508 3411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.938587 3411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.939651 3411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.975065 3411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:29.975816 3411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:30.013684 3411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:30.045423 3411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:30.084614 3411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:30.124469 3411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:30.166052 3411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:30.172421 3411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:30.214838 3411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
I20260804 14:53:30.255261 3770 raft_consensus.cc:482] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260804 14:53:30.255555 3770 raft_consensus.cc:504] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 } } peers { permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 } } peers { permanent_uuid: "9a7053094ff74e82b9a176d1db50adca" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40315 } attrs { } }
W20260804 14:53:30.256928 3411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
I20260804 14:53:30.257181 3770 leader_election.cc:288] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dc0d993490e247b9a6aa70eedc63eb06 (127.24.6.130:41505), 9a7053094ff74e82b9a176d1db50adca (127.24.6.129:40315)
W20260804 14:53:30.258077 3378 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.6.130:41505: connect: Connection refused (error 111)
W20260804 14:53:30.258348 3411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51898: Illegal state: replica 422632429a694bde88205930ec518cc6 is not leader of this config: current role FOLLOWER
W20260804 14:53:30.259418 3378 leader_election.cc:334] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer dc0d993490e247b9a6aa70eedc63eb06 (127.24.6.130:41505): Network error: Client connection negotiation failed: client connection to 127.24.6.130:41505: connect: Connection refused (error 111)
I20260804 14:53:30.262852 3677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e62bf4b11eab47f193f28d71e8bd8bdf" candidate_uuid: "422632429a694bde88205930ec518cc6" candidate_term: 3 candidate_status { last_received { term: 2 index: 41 } } ignore_live_leader: false dest_uuid: "9a7053094ff74e82b9a176d1db50adca" is_pre_election: true
I20260804 14:53:30.264230 3677 raft_consensus.cc:1789] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 1.1
I20260804 14:53:30.264695 3677 raft_consensus.cc:2454] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 422632429a694bde88205930ec518cc6 in term 2.
I20260804 14:53:30.265252 3374 leader_election.cc:302] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 422632429a694bde88205930ec518cc6, 9a7053094ff74e82b9a176d1db50adca; no voters: dc0d993490e247b9a6aa70eedc63eb06
I20260804 14:53:30.265619 3770 raft_consensus.cc:2781] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260804 14:53:30.265735 3770 raft_consensus.cc:482] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260804 14:53:30.265777 3770 raft_consensus.cc:3037] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [term 2 FOLLOWER]: Advancing to term 3
I20260804 14:53:30.268558 3770 raft_consensus.cc:504] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 } } peers { permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 } } peers { permanent_uuid: "9a7053094ff74e82b9a176d1db50adca" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40315 } attrs { } }
I20260804 14:53:30.268997 3770 leader_election.cc:288] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [CANDIDATE]: Term 3 election: Requested vote from peers dc0d993490e247b9a6aa70eedc63eb06 (127.24.6.130:41505), 9a7053094ff74e82b9a176d1db50adca (127.24.6.129:40315)
I20260804 14:53:30.269564 3677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e62bf4b11eab47f193f28d71e8bd8bdf" candidate_uuid: "422632429a694bde88205930ec518cc6" candidate_term: 3 candidate_status { last_received { term: 2 index: 41 } } ignore_live_leader: false dest_uuid: "9a7053094ff74e82b9a176d1db50adca"
I20260804 14:53:30.269707 3677 raft_consensus.cc:1789] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 1.1
I20260804 14:53:30.269774 3677 raft_consensus.cc:3037] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [term 2 FOLLOWER]: Advancing to term 3
W20260804 14:53:30.270615 3378 leader_election.cc:334] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer dc0d993490e247b9a6aa70eedc63eb06 (127.24.6.130:41505): Network error: Client connection negotiation failed: client connection to 127.24.6.130:41505: connect: Connection refused (error 111)
I20260804 14:53:30.271186 3677 raft_consensus.cc:2454] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 422632429a694bde88205930ec518cc6 in term 3.
I20260804 14:53:30.271574 3374 leader_election.cc:302] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 422632429a694bde88205930ec518cc6, 9a7053094ff74e82b9a176d1db50adca; no voters: dc0d993490e247b9a6aa70eedc63eb06
I20260804 14:53:30.271749 3770 raft_consensus.cc:2781] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [term 3 FOLLOWER]: Leader election won for term 3
I20260804 14:53:30.271996 3770 raft_consensus.cc:686] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [term 3 LEADER]: Becoming Leader. State: Replica: 422632429a694bde88205930ec518cc6, State: Running, Role: LEADER
I20260804 14:53:30.272400 3770 consensus_queue.cc:237] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 2.41, Last appended by leader: 41, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 } } peers { permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 } } peers { permanent_uuid: "9a7053094ff74e82b9a176d1db50adca" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40315 } attrs { } }
I20260804 14:53:30.275792 3261 catalog_manager.cc:5495] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "422632429a694bde88205930ec518cc6" committed_config { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 } } peers { permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 } } peers { permanent_uuid: "9a7053094ff74e82b9a176d1db50adca" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40315 } attrs { } } }
I20260804 14:53:30.279873 3749 heartbeater.cc:503] Master 127.24.6.190:33693 was elected leader, sending a full tablet report...
W20260804 14:53:30.308557 3374 consensus_peers.cc:487] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 -> Peer 9a7053094ff74e82b9a176d1db50adca (127.24.6.129:40315): Couldn't send request to peer 9a7053094ff74e82b9a176d1db50adca. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260804 14:53:30.308583 3378 consensus_peers.cc:487] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6 -> Peer dc0d993490e247b9a6aa70eedc63eb06 (127.24.6.130:41505): Couldn't send request to peer dc0d993490e247b9a6aa70eedc63eb06. Status: Network error: Client connection negotiation failed: client connection to 127.24.6.130:41505: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260804 14:53:30.332027 3756 meta_cache.cc:302] tablet e62bf4b11eab47f193f28d71e8bd8bdf: replica dc0d993490e247b9a6aa70eedc63eb06 (127.24.6.130:41505) has failed: Network error: Client connection negotiation failed: client connection to 127.24.6.130:41505: connect: Connection refused (error 111)
I20260804 14:53:30.917236 3788 tablet_replica.cc:330] stopping tablet replica
I20260804 14:53:30.917534 3788 raft_consensus.cc:2229] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [term 3 FOLLOWER]: Raft consensus shutting down.
I20260804 14:53:30.917672 3788 raft_consensus.cc:2258] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [term 3 FOLLOWER]: Raft consensus is shut down!
I20260804 14:53:30.917936 3788 ts_tablet_manager.cc:927] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca: Initiating tablet copy from peer 422632429a694bde88205930ec518cc6 (127.24.6.131:41365)
I20260804 14:53:30.918618 3788 tablet_copy_client.cc:316] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca: tablet copy: Beginning tablet copy session from remote peer at address 127.24.6.131:41365
I20260804 14:53:30.923502 3507 tablet_copy_service.cc:140] P 422632429a694bde88205930ec518cc6: Received BeginTabletCopySession request for tablet e62bf4b11eab47f193f28d71e8bd8bdf from peer 9a7053094ff74e82b9a176d1db50adca ({username='slave'} at 127.24.6.129:46619)
I20260804 14:53:30.923700 3507 tablet_copy_service.cc:161] P 422632429a694bde88205930ec518cc6: Beginning new tablet copy session on tablet e62bf4b11eab47f193f28d71e8bd8bdf from peer 9a7053094ff74e82b9a176d1db50adca at {username='slave'} at 127.24.6.129:46619: session id = 9a7053094ff74e82b9a176d1db50adca-e62bf4b11eab47f193f28d71e8bd8bdf
I20260804 14:53:30.925360 3507 tablet_copy_source_session.cc:205] T e62bf4b11eab47f193f28d71e8bd8bdf P 422632429a694bde88205930ec518cc6: Tablet Copy: opened 5 blocks and 3 log segments
I20260804 14:53:30.927842 3788 ts_tablet_manager.cc:1905] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260804 14:53:30.930287 3788 ts_tablet_manager.cc:1918] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1
I20260804 14:53:30.930708 3788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e62bf4b11eab47f193f28d71e8bd8bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:30.932857 3788 tablet_copy_client.cc:791] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca: tablet copy: Starting download of 5 data blocks...
I20260804 14:53:30.943276 3788 tablet_copy_client.cc:655] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca: tablet copy: Starting download of 3 WAL segments...
I20260804 14:53:30.952687 3788 tablet_copy_client.cc:531] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260804 14:53:30.955142 3788 tablet_bootstrap.cc:492] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca: Bootstrap starting.
I20260804 14:53:30.963523 3788 log.cc:824] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:30.971169 3788 tablet_bootstrap.cc:492] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca: Bootstrap replayed 1/3 log segments. Stats: ops{read=7 overwritten=0 applied=5 ignored=5} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20260804 14:53:30.975979 3788 tablet_bootstrap.cc:492] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca: Bootstrap replayed 2/3 log segments. Stats: ops{read=14 overwritten=0 applied=13 ignored=13} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260804 14:53:30.979008 3788 tablet_bootstrap.cc:492] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca: Bootstrap replayed 3/3 log segments. Stats: ops{read=23 overwritten=0 applied=18 ignored=16} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 5 replicates
I20260804 14:53:30.979590 3788 tablet_bootstrap.cc:492] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca: Bootstrap complete.
I20260804 14:53:30.980465 3788 ts_tablet_manager.cc:1397] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca: Time spent bootstrapping tablet: real 0.025s user 0.019s sys 0.008s
I20260804 14:53:30.982437 3788 raft_consensus.cc:348] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [term 3 FOLLOWER]: Replica starting. Triggering 5 pending ops. Active config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 } } peers { permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 } } peers { permanent_uuid: "9a7053094ff74e82b9a176d1db50adca" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40315 } attrs { } }
I20260804 14:53:30.984397 3788 raft_consensus.cc:729] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a7053094ff74e82b9a176d1db50adca, State: Initialized, Role: FOLLOWER
I20260804 14:53:30.984918 3788 consensus_queue.cc:260] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 41, Last appended: 3.46, Last appended by leader: 46, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "422632429a694bde88205930ec518cc6" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 41365 } } peers { permanent_uuid: "dc0d993490e247b9a6aa70eedc63eb06" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 41505 } } peers { permanent_uuid: "9a7053094ff74e82b9a176d1db50adca" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 40315 } attrs { } }
I20260804 14:53:30.985918 3788 ts_tablet_manager.cc:1428] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260804 14:53:30.986848 3505 tablet_copy_service.cc:342] P 422632429a694bde88205930ec518cc6: Request end of tablet copy session 9a7053094ff74e82b9a176d1db50adca-e62bf4b11eab47f193f28d71e8bd8bdf received from {username='slave'} at 127.24.6.129:46619
I20260804 14:53:30.986990 3505 tablet_copy_service.cc:433] P 422632429a694bde88205930ec518cc6: ending tablet copy session 9a7053094ff74e82b9a176d1db50adca-e62bf4b11eab47f193f28d71e8bd8bdf on tablet e62bf4b11eab47f193f28d71e8bd8bdf with peer 9a7053094ff74e82b9a176d1db50adca
I20260804 14:53:31.066642 3750 maintenance_manager.cc:419] P 9a7053094ff74e82b9a176d1db50adca: Scheduling UndoDeltaBlockGCOp(e62bf4b11eab47f193f28d71e8bd8bdf): 551 bytes on disk
I20260804 14:53:31.067974 3576 maintenance_manager.cc:643] P 9a7053094ff74e82b9a176d1db50adca: UndoDeltaBlockGCOp(e62bf4b11eab47f193f28d71e8bd8bdf) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20260804 14:53:31.488000 3677 raft_consensus.cc:1203] T e62bf4b11eab47f193f28d71e8bd8bdf P 9a7053094ff74e82b9a176d1db50adca [term 3 FOLLOWER]: Deduplicated request from leader. Original: 2.41->[3.42-3.46] Dedup: 3.46->[]
I20260804 14:53:31.491436 3806 mvcc.cc:205] Tried to move back new op lower bound from 7314862942359150592 to 7314862941278613504. Current Snapshot: MvccSnapshot[applied={T|T < 7314862941415096320}]
I20260804 14:53:31.491297 3809 mvcc.cc:205] Tried to move back new op lower bound from 7314862941948575744 to 7314862941278613504. Current Snapshot: MvccSnapshot[applied={T|T < 7314862941415096320}]
I20260804 14:53:31.493912 3810 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:51898 (ReqId={client: 087daea1495449baaac18a5d278b5426, seq_no=1, attempt_no=41}) took 1116 ms. Trace:
I20260804 14:53:31.494107 3810 rpcz_store.cc:276] 0804 14:53:30.377868 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:53:30.377967 (+ 99us) service_pool.cc:227] Handling call
0804 14:53:31.493892 (+1115925us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:53:30.378540 (+ 0us) write_op.cc:183] PREPARE: starting on tablet e62bf4b11eab47f193f28d71e8bd8bdf
0804 14:53:30.378637 (+ 97us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:53:30.378642 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:53:30.378643 (+ 1us) tablet.cc:660] Decoding operations
0804 14:53:30.378860 (+ 217us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:53:30.378897 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0804 14:53:30.378900 (+ 3us) tablet.cc:683] Acquiring locks for 50 operations
0804 14:53:30.379087 (+ 187us) tablet.cc:699] Row locks acquired
0804 14:53:30.379089 (+ 2us) write_op.cc:260] PREPARE: finished
0804 14:53:30.379138 (+ 49us) write_op.cc:270] Start()
0804 14:53:30.379168 (+ 30us) write_op.cc:276] Timestamp: P: 1785855210379074 usec, L: 0
0804 14:53:30.379170 (+ 2us) op_driver.cc:331] REPLICATION: starting
0804 14:53:30.379354 (+ 184us) log.cc:842] Serialized 2455 byte log entry
0804 14:53:31.490347 (+1110993us) op_driver.cc:444] REPLICATION: finished
0804 14:53:31.490633 (+ 286us) write_op.cc:301] APPLY: starting
0804 14:53:31.490649 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0804 14:53:31.491931 (+ 1282us) tablet.cc:1368] finished BulkCheckPresence
0804 14:53:31.491949 (+ 18us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:53:31.493091 (+ 1142us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:53:31.493273 (+ 182us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=40,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:53:31.493296 (+ 23us) write_op.cc:312] APPLY: finished
0804 14:53:31.493496 (+ 200us) log.cc:842] Serialized 319 byte log entry
0804 14:53:31.493605 (+ 109us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:53:31.493701 (+ 96us) write_op.cc:454] Released schema lock
0804 14:53:31.493722 (+ 21us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":240,"cfile_cache_miss":2,"cfile_cache_miss_bytes":4127,"cfile_init":1,"lbm_read_time_us":155,"lbm_reads_lt_1ms":6,"num_ops":50,"prepare.queue_time_us":73,"prepare.run_cpu_time_us":980,"prepare.run_wall_time_us":980,"raft.queue_time_us":189,"raft.run_cpu_time_us":14,"raft.run_wall_time_us":16,"replication_time_us":1111162,"spinlock_wait_cycles":768,"thread_start_us":131,"threads_started":1}]]}
I20260804 14:53:31.496256 3807 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:51898 (ReqId={client: 087daea1495449baaac18a5d278b5426, seq_no=0, attempt_no=40}) took 1192 ms. Trace:
I20260804 14:53:31.496415 3807 rpcz_store.cc:276] 0804 14:53:30.303608 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:53:30.303678 (+ 70us) service_pool.cc:227] Handling call
0804 14:53:31.496242 (+1192564us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:53:30.304423 (+ 0us) write_op.cc:183] PREPARE: starting on tablet e62bf4b11eab47f193f28d71e8bd8bdf
0804 14:53:30.304556 (+ 133us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:53:30.304584 (+ 28us) write_op.cc:435] Acquired schema lock
0804 14:53:30.304585 (+ 1us) tablet.cc:660] Decoding operations
0804 14:53:30.305465 (+ 880us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:53:30.305524 (+ 59us) write_op.cc:641] Partition lock acquired for write op
0804 14:53:30.305530 (+ 6us) tablet.cc:683] Acquiring locks for 50 operations
0804 14:53:30.306407 (+ 877us) tablet.cc:699] Row locks acquired
0804 14:53:30.306409 (+ 2us) write_op.cc:260] PREPARE: finished
0804 14:53:30.306479 (+ 70us) write_op.cc:270] Start()
0804 14:53:30.306520 (+ 41us) write_op.cc:276] Timestamp: P: 1785855210306420 usec, L: 0
0804 14:53:30.306524 (+ 4us) op_driver.cc:331] REPLICATION: starting
0804 14:53:30.306769 (+ 245us) log.cc:842] Serialized 2455 byte log entry
0804 14:53:31.490191 (+1183422us) op_driver.cc:444] REPLICATION: finished
0804 14:53:31.490487 (+ 296us) write_op.cc:301] APPLY: starting
0804 14:53:31.490515 (+ 28us) tablet.cc:1365] starting BulkCheckPresence
0804 14:53:31.491914 (+ 1399us) tablet.cc:1368] finished BulkCheckPresence
0804 14:53:31.491930 (+ 16us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:53:31.493229 (+ 1299us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:53:31.493273 (+ 44us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=39,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:53:31.493296 (+ 23us) write_op.cc:312] APPLY: finished
0804 14:53:31.495870 (+ 2574us) log.cc:842] Serialized 319 byte log entry
0804 14:53:31.496132 (+ 262us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:53:31.496155 (+ 23us) write_op.cc:454] Released schema lock
0804 14:53:31.496163 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":175,"cfile_cache_miss":2,"cfile_cache_miss_bytes":4127,"lbm_read_time_us":42,"lbm_reads_lt_1ms":2,"mutex_wait_us":56,"num_ops":50,"prepare.queue_time_us":302,"prepare.run_cpu_time_us":2563,"prepare.run_wall_time_us":2563,"raft.queue_time_us":139,"raft.run_cpu_time_us":779,"raft.run_wall_time_us":779,"replication_time_us":1183612,"spinlock_wait_cycles":402944,"thread_start_us":219,"threads_started":2}]]}
I20260804 14:53:31.498970 3807 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:51898 (ReqId={client: 087daea1495449baaac18a5d278b5426, seq_no=3, attempt_no=43}) took 1063 ms. Trace:
I20260804 14:53:31.499109 3807 rpcz_store.cc:276] 0804 14:53:30.435422 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:53:30.435508 (+ 86us) service_pool.cc:227] Handling call
0804 14:53:31.498956 (+1063448us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:53:30.436038 (+ 0us) write_op.cc:183] PREPARE: starting on tablet e62bf4b11eab47f193f28d71e8bd8bdf
0804 14:53:30.436134 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:53:30.436139 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:53:30.436140 (+ 1us) tablet.cc:660] Decoding operations
0804 14:53:30.436400 (+ 260us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:53:30.436411 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0804 14:53:30.436414 (+ 3us) tablet.cc:683] Acquiring locks for 50 operations
0804 14:53:30.436684 (+ 270us) tablet.cc:699] Row locks acquired
0804 14:53:30.436685 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:53:30.436732 (+ 47us) write_op.cc:270] Start()
0804 14:53:30.436774 (+ 42us) write_op.cc:276] Timestamp: P: 1785855210436664 usec, L: 0
0804 14:53:30.436775 (+ 1us) op_driver.cc:331] REPLICATION: starting
0804 14:53:30.436935 (+ 160us) log.cc:842] Serialized 2455 byte log entry
0804 14:53:31.490542 (+1053607us) op_driver.cc:444] REPLICATION: finished
0804 14:53:31.497578 (+ 7036us) write_op.cc:301] APPLY: starting
0804 14:53:31.497592 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0804 14:53:31.497801 (+ 209us) tablet.cc:1368] finished BulkCheckPresence
0804 14:53:31.497813 (+ 12us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:53:31.498473 (+ 660us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:53:31.498529 (+ 56us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=43,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:53:31.498534 (+ 5us) write_op.cc:312] APPLY: finished
0804 14:53:31.498649 (+ 115us) log.cc:842] Serialized 319 byte log entry
0804 14:53:31.498853 (+ 204us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:53:31.498876 (+ 23us) write_op.cc:454] Released schema lock
0804 14:53:31.498884 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":6999,"num_ops":50,"prepare.queue_time_us":85,"prepare.run_cpu_time_us":1099,"prepare.run_wall_time_us":1170,"raft.queue_time_us":132,"raft.run_cpu_time_us":17,"raft.run_wall_time_us":21,"replication_time_us":1053750,"thread_start_us":82,"threads_started":1}]]}
I20260804 14:53:32.314893 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 3548
I20260804 14:53:32.327293 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 3317
I20260804 14:53:32.342763 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 3218
2026-08-04T14:53:32Z chronyd exiting
[ OK ] RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy (9699 ms)
[ RUN ] RaftConsensusElectionITest.DisruptiveServerAndSlowWAL
I20260804 14:53:32.373193 24602 test_util.cc:268] Using random seed: -779238206
I20260804 14:53:32.374105 24602 ts_itest-base.cc:112] Starting cluster with:
I20260804 14:53:32.374161 24602 ts_itest-base.cc:113] --------------
I20260804 14:53:32.374183 24602 ts_itest-base.cc:114] 3 tablet servers
I20260804 14:53:32.374212 24602 ts_itest-base.cc:115] 3 replicas per TS
I20260804 14:53:32.374238 24602 ts_itest-base.cc:116] --------------
2026-08-04T14:53:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-04T14:53:32Z Disabled control of system clock
I20260804 14:53:32.387527 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.6.190:35759
--webserver_interface=127.24.6.190
--webserver_port=0
--builtin_ntp_servers=127.24.6.148:36223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.6.190:35759 with env {}
W20260804 14:53:32.525694 3834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:32.526077 3834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:32.526145 3834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:32.530614 3834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260804 14:53:32.530711 3834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:32.530743 3834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260804 14:53:32.530781 3834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260804 14:53:32.535981 3834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:36223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.6.190:35759
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.6.190:35759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.24.6.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.3834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:32.537492 3834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:32.539054 3834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:32.548676 3843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:32.548691 3842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:32.548691 3845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:32.548767 3834 server_base.cc:1034] running on GCE node
I20260804 14:53:32.549716 3834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:32.550971 3834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:32.552196 3834 hybrid_clock.cc:648] HybridClock initialized: now 1785855212552159 us; error 86 us; skew 500 ppm
I20260804 14:53:32.558800 3834 webserver.cc:460] Webserver started at http://127.24.6.190:38749/ using document root <none> and password file <none>
I20260804 14:53:32.559587 3834 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:32.559695 3834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:32.559963 3834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:32.562132 3834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "19b5c4b8d9fd4643a375ef3c598ed4bd"
format_stamp: "Formatted at 2026-08-04 14:53:32 on dist-test-slave-92m1"
I20260804 14:53:32.562758 3834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "19b5c4b8d9fd4643a375ef3c598ed4bd"
format_stamp: "Formatted at 2026-08-04 14:53:32 on dist-test-slave-92m1"
I20260804 14:53:32.566952 3834 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260804 14:53:32.569907 3858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:32.571341 3834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260804 14:53:32.571516 3834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
uuid: "19b5c4b8d9fd4643a375ef3c598ed4bd"
format_stamp: "Formatted at 2026-08-04 14:53:32 on dist-test-slave-92m1"
I20260804 14:53:32.571640 3834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:32.590390 3834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:32.591174 3834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:32.591445 3834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:32.598922 3834 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.190:35759
I20260804 14:53:32.598932 3935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.190:35759 every 8 connection(s)
I20260804 14:53:32.600015 3834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260804 14:53:32.603525 3936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:32.608819 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 3834
I20260804 14:53:32.608985 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260804 14:53:32.609012 3936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19b5c4b8d9fd4643a375ef3c598ed4bd: Bootstrap starting.
I20260804 14:53:32.611816 3936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19b5c4b8d9fd4643a375ef3c598ed4bd: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:32.612695 3936 log.cc:824] T 00000000000000000000000000000000 P 19b5c4b8d9fd4643a375ef3c598ed4bd: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:32.614703 3936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19b5c4b8d9fd4643a375ef3c598ed4bd: No bootstrap required, opened a new log
I20260804 14:53:32.619040 3936 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19b5c4b8d9fd4643a375ef3c598ed4bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19b5c4b8d9fd4643a375ef3c598ed4bd" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 35759 } }
I20260804 14:53:32.619333 3936 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19b5c4b8d9fd4643a375ef3c598ed4bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:53:32.619406 3936 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19b5c4b8d9fd4643a375ef3c598ed4bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19b5c4b8d9fd4643a375ef3c598ed4bd, State: Initialized, Role: FOLLOWER
I20260804 14:53:32.619868 3936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19b5c4b8d9fd4643a375ef3c598ed4bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19b5c4b8d9fd4643a375ef3c598ed4bd" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 35759 } }
I20260804 14:53:32.619992 3936 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19b5c4b8d9fd4643a375ef3c598ed4bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260804 14:53:32.620088 3936 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19b5c4b8d9fd4643a375ef3c598ed4bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260804 14:53:32.620204 3936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19b5c4b8d9fd4643a375ef3c598ed4bd [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:32.621205 3936 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19b5c4b8d9fd4643a375ef3c598ed4bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19b5c4b8d9fd4643a375ef3c598ed4bd" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 35759 } }
I20260804 14:53:32.621547 3936 leader_election.cc:302] T 00000000000000000000000000000000 P 19b5c4b8d9fd4643a375ef3c598ed4bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19b5c4b8d9fd4643a375ef3c598ed4bd; no voters:
I20260804 14:53:32.621874 3936 leader_election.cc:288] T 00000000000000000000000000000000 P 19b5c4b8d9fd4643a375ef3c598ed4bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260804 14:53:32.622007 3943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19b5c4b8d9fd4643a375ef3c598ed4bd [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:32.622279 3943 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19b5c4b8d9fd4643a375ef3c598ed4bd [term 1 LEADER]: Becoming Leader. State: Replica: 19b5c4b8d9fd4643a375ef3c598ed4bd, State: Running, Role: LEADER
I20260804 14:53:32.622608 3943 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19b5c4b8d9fd4643a375ef3c598ed4bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19b5c4b8d9fd4643a375ef3c598ed4bd" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 35759 } }
I20260804 14:53:32.623186 3936 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19b5c4b8d9fd4643a375ef3c598ed4bd [sys.catalog]: configured and running, proceeding with master startup.
I20260804 14:53:32.624351 3948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19b5c4b8d9fd4643a375ef3c598ed4bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19b5c4b8d9fd4643a375ef3c598ed4bd. Latest consensus state: current_term: 1 leader_uuid: "19b5c4b8d9fd4643a375ef3c598ed4bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19b5c4b8d9fd4643a375ef3c598ed4bd" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 35759 } } }
I20260804 14:53:32.624500 3948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19b5c4b8d9fd4643a375ef3c598ed4bd [sys.catalog]: This master's current role is: LEADER
I20260804 14:53:32.624821 3947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19b5c4b8d9fd4643a375ef3c598ed4bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19b5c4b8d9fd4643a375ef3c598ed4bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19b5c4b8d9fd4643a375ef3c598ed4bd" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 35759 } } }
I20260804 14:53:32.624933 3947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19b5c4b8d9fd4643a375ef3c598ed4bd [sys.catalog]: This master's current role is: LEADER
I20260804 14:53:32.625049 3956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260804 14:53:32.628012 3956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260804 14:53:32.634554 3956 catalog_manager.cc:1347] Generated new cluster ID: 25885227c3344c0eba73d8ec856a6c94
I20260804 14:53:32.634678 3956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260804 14:53:32.656239 3956 catalog_manager.cc:1370] Generated new certificate authority record
I20260804 14:53:32.657316 3956 catalog_manager.cc:1504] Loading token signing keys...
I20260804 14:53:32.663151 3956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19b5c4b8d9fd4643a375ef3c598ed4bd: Generated new TSK 0
I20260804 14:53:32.663983 3956 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260804 14:53:32.669390 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.129:0
--local_ip_for_outbound_sockets=127.24.6.129
--webserver_interface=127.24.6.129
--webserver_port=0
--tserver_master_addrs=127.24.6.190:35759
--builtin_ntp_servers=127.24.6.148:36223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--leader_failure_max_missed_heartbeat_periods=3 with env {}
W20260804 14:53:32.807688 3970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:32.808076 3970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:32.808132 3970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:32.812466 3970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:32.812587 3970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.129
I20260804 14:53:32.817628 3970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:36223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.24.6.129
--webserver_port=0
--tserver_master_addrs=127.24.6.190:35759
--never_fsync=true
--heap_profile_path=/tmp/kudu.3970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.129
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:32.818979 3970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:32.820518 3970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260804 14:53:32.831697 3970 server_base.cc:1034] running on GCE node
W20260804 14:53:32.831635 3978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:32.831635 3975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:32.831638 3976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:32.832401 3970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:32.833204 3970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:32.834385 3970 hybrid_clock.cc:648] HybridClock initialized: now 1785855212834351 us; error 60 us; skew 500 ppm
I20260804 14:53:32.836740 3970 webserver.cc:460] Webserver started at http://127.24.6.129:34653/ using document root <none> and password file <none>
I20260804 14:53:32.837523 3970 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:32.837620 3970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:32.837877 3970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:32.840315 3970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d820c59a173343aea38e08212164213d"
format_stamp: "Formatted at 2026-08-04 14:53:32 on dist-test-slave-92m1"
I20260804 14:53:32.840996 3970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d820c59a173343aea38e08212164213d"
format_stamp: "Formatted at 2026-08-04 14:53:32 on dist-test-slave-92m1"
I20260804 14:53:32.845427 3970 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260804 14:53:32.848393 3987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:32.849900 3970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260804 14:53:32.850072 3970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d820c59a173343aea38e08212164213d"
format_stamp: "Formatted at 2026-08-04 14:53:32 on dist-test-slave-92m1"
I20260804 14:53:32.850229 3970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:32.873924 3970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:32.874846 3970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:32.875102 3970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:32.875902 3970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:53:32.877120 3970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:53:32.877192 3970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:32.877262 3970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:53:32.877308 3970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:32.887143 3970 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.129:36711
I20260804 14:53:32.887168 4147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.129:36711 every 8 connection(s)
I20260804 14:53:32.888242 3970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260804 14:53:32.889640 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 3970
I20260804 14:53:32.889779 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260804 14:53:32.892762 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.130:0
--local_ip_for_outbound_sockets=127.24.6.130
--webserver_interface=127.24.6.130
--webserver_port=0
--tserver_master_addrs=127.24.6.190:35759
--builtin_ntp_servers=127.24.6.148:36223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--leader_failure_max_missed_heartbeat_periods=3 with env {}
I20260804 14:53:32.903309 4149 heartbeater.cc:347] Connected to a master server at 127.24.6.190:35759
I20260804 14:53:32.903626 4149 heartbeater.cc:464] Registering TS with master...
I20260804 14:53:32.904248 4149 heartbeater.cc:511] Master 127.24.6.190:35759 requested a full tablet report, sending...
I20260804 14:53:32.905952 3884 ts_manager.cc:194] Registered new tserver with Master: d820c59a173343aea38e08212164213d (127.24.6.129:36711)
I20260804 14:53:32.907298 3884 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.129:34013
W20260804 14:53:33.021251 4153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:33.021669 4153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:33.021755 4153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:33.027029 4153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:33.027257 4153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.130
I20260804 14:53:33.033386 4153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:36223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.24.6.130
--webserver_port=0
--tserver_master_addrs=127.24.6.190:35759
--never_fsync=true
--heap_profile_path=/tmp/kudu.4153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.130
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:33.034850 4153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:33.036888 4153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:33.049316 4163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:33.049386 4153 server_base.cc:1034] running on GCE node
W20260804 14:53:33.049453 4160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:33.049327 4161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:33.050148 4153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:33.051028 4153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:33.052285 4153 hybrid_clock.cc:648] HybridClock initialized: now 1785855213052240 us; error 56 us; skew 500 ppm
I20260804 14:53:33.055122 4153 webserver.cc:460] Webserver started at http://127.24.6.130:34355/ using document root <none> and password file <none>
I20260804 14:53:33.055970 4153 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:33.056104 4153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:33.056406 4153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:33.058910 4153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "113ef42699a14b8698690b425cfdb53a"
format_stamp: "Formatted at 2026-08-04 14:53:33 on dist-test-slave-92m1"
I20260804 14:53:33.059763 4153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "113ef42699a14b8698690b425cfdb53a"
format_stamp: "Formatted at 2026-08-04 14:53:33 on dist-test-slave-92m1"
I20260804 14:53:33.064985 4153 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260804 14:53:33.068292 4175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:33.070075 4153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260804 14:53:33.070273 4153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "113ef42699a14b8698690b425cfdb53a"
format_stamp: "Formatted at 2026-08-04 14:53:33 on dist-test-slave-92m1"
I20260804 14:53:33.070422 4153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:33.092528 4153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:33.093420 4153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:33.093652 4153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:33.094347 4153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:53:33.095500 4153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:53:33.095572 4153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:33.095649 4153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:53:33.095690 4153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:33.105240 4153 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.130:38597
I20260804 14:53:33.105262 4348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.130:38597 every 8 connection(s)
I20260804 14:53:33.106242 4153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260804 14:53:33.113377 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 4153
I20260804 14:53:33.113497 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260804 14:53:33.116215 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.131:0
--local_ip_for_outbound_sockets=127.24.6.131
--webserver_interface=127.24.6.131
--webserver_port=0
--tserver_master_addrs=127.24.6.190:35759
--builtin_ntp_servers=127.24.6.148:36223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--leader_failure_max_missed_heartbeat_periods=3 with env {}
I20260804 14:53:33.118716 4350 heartbeater.cc:347] Connected to a master server at 127.24.6.190:35759
I20260804 14:53:33.119009 4350 heartbeater.cc:464] Registering TS with master...
I20260804 14:53:33.119673 4350 heartbeater.cc:511] Master 127.24.6.190:35759 requested a full tablet report, sending...
I20260804 14:53:33.121016 3884 ts_manager.cc:194] Registered new tserver with Master: 113ef42699a14b8698690b425cfdb53a (127.24.6.130:38597)
I20260804 14:53:33.121915 3884 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.130:35327
W20260804 14:53:33.249012 4353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:33.249333 4353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:33.249372 4353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:33.253666 4353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:33.253789 4353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.131
I20260804 14:53:33.258987 4353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:36223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.24.6.131
--webserver_port=0
--tserver_master_addrs=127.24.6.190:35759
--never_fsync=true
--heap_profile_path=/tmp/kudu.4353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.131
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:33.260327 4353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:33.262578 4353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:33.273963 4361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:33.273998 4363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:33.274212 4353 server_base.cc:1034] running on GCE node
W20260804 14:53:33.274012 4358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:33.274919 4353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:33.275915 4353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:33.277145 4353 hybrid_clock.cc:648] HybridClock initialized: now 1785855213277099 us; error 66 us; skew 500 ppm
I20260804 14:53:33.279743 4353 webserver.cc:460] Webserver started at http://127.24.6.131:34235/ using document root <none> and password file <none>
I20260804 14:53:33.280447 4353 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:33.280556 4353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:33.280808 4353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:33.285768 4353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "890cbe5f30df441dadefc35f3984b431"
format_stamp: "Formatted at 2026-08-04 14:53:33 on dist-test-slave-92m1"
I20260804 14:53:33.286527 4353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "890cbe5f30df441dadefc35f3984b431"
format_stamp: "Formatted at 2026-08-04 14:53:33 on dist-test-slave-92m1"
I20260804 14:53:33.291100 4353 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260804 14:53:33.294196 4370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:33.295805 4353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260804 14:53:33.296047 4353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "890cbe5f30df441dadefc35f3984b431"
format_stamp: "Formatted at 2026-08-04 14:53:33 on dist-test-slave-92m1"
I20260804 14:53:33.296218 4353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:33.313560 4353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:33.314402 4353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:33.314632 4353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:33.315346 4353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:53:33.316476 4353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:53:33.316543 4353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:33.316615 4353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:53:33.316660 4353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:33.326443 4353 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.131:44351
I20260804 14:53:33.326474 4547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.131:44351 every 8 connection(s)
I20260804 14:53:33.327642 4353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260804 14:53:33.337023 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 4353
I20260804 14:53:33.337195 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260804 14:53:33.339874 4548 heartbeater.cc:347] Connected to a master server at 127.24.6.190:35759
I20260804 14:53:33.340263 4548 heartbeater.cc:464] Registering TS with master...
I20260804 14:53:33.340857 4548 heartbeater.cc:511] Master 127.24.6.190:35759 requested a full tablet report, sending...
I20260804 14:53:33.342006 3883 ts_manager.cc:194] Registered new tserver with Master: 890cbe5f30df441dadefc35f3984b431 (127.24.6.131:44351)
I20260804 14:53:33.342713 3883 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.131:34837
I20260804 14:53:33.352699 24602 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260804 14:53:33.364359 3883 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50260:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20260804 14:53:33.367658 3883 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260804 14:53:33.387252 4249 tablet_service.cc:1467] Processing CreateTablet for tablet 525ce34640de48998c33bef28893ab50 (DEFAULT_TABLE table=TestTable [id=2d60426558204d938c6f85781dbf7595]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:53:33.387436 4452 tablet_service.cc:1467] Processing CreateTablet for tablet 525ce34640de48998c33bef28893ab50 (DEFAULT_TABLE table=TestTable [id=2d60426558204d938c6f85781dbf7595]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:53:33.387812 4056 tablet_service.cc:1467] Processing CreateTablet for tablet 525ce34640de48998c33bef28893ab50 (DEFAULT_TABLE table=TestTable [id=2d60426558204d938c6f85781dbf7595]), partition=RANGE (key) PARTITION UNBOUNDED
I20260804 14:53:33.388479 4249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 525ce34640de48998c33bef28893ab50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:33.388579 4452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 525ce34640de48998c33bef28893ab50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:33.388759 4056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 525ce34640de48998c33bef28893ab50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:33.396004 4574 tablet_bootstrap.cc:492] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a: Bootstrap starting.
I20260804 14:53:33.396334 4576 tablet_bootstrap.cc:492] T 525ce34640de48998c33bef28893ab50 P 890cbe5f30df441dadefc35f3984b431: Bootstrap starting.
I20260804 14:53:33.396497 4575 tablet_bootstrap.cc:492] T 525ce34640de48998c33bef28893ab50 P d820c59a173343aea38e08212164213d: Bootstrap starting.
I20260804 14:53:33.398365 4574 tablet_bootstrap.cc:654] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:33.398633 4576 tablet_bootstrap.cc:654] T 525ce34640de48998c33bef28893ab50 P 890cbe5f30df441dadefc35f3984b431: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:33.398633 4575 tablet_bootstrap.cc:654] T 525ce34640de48998c33bef28893ab50 P d820c59a173343aea38e08212164213d: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:33.399333 4574 log.cc:824] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:33.399533 4576 log.cc:824] T 525ce34640de48998c33bef28893ab50 P 890cbe5f30df441dadefc35f3984b431: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:33.399573 4575 log.cc:824] T 525ce34640de48998c33bef28893ab50 P d820c59a173343aea38e08212164213d: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:33.401728 4574 tablet_bootstrap.cc:492] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a: No bootstrap required, opened a new log
I20260804 14:53:33.401728 4576 tablet_bootstrap.cc:492] T 525ce34640de48998c33bef28893ab50 P 890cbe5f30df441dadefc35f3984b431: No bootstrap required, opened a new log
I20260804 14:53:33.401959 4574 ts_tablet_manager.cc:1397] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260804 14:53:33.401959 4576 ts_tablet_manager.cc:1397] T 525ce34640de48998c33bef28893ab50 P 890cbe5f30df441dadefc35f3984b431: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260804 14:53:33.401986 4575 tablet_bootstrap.cc:492] T 525ce34640de48998c33bef28893ab50 P d820c59a173343aea38e08212164213d: No bootstrap required, opened a new log
I20260804 14:53:33.402200 4575 ts_tablet_manager.cc:1397] T 525ce34640de48998c33bef28893ab50 P d820c59a173343aea38e08212164213d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260804 14:53:33.405911 4575 raft_consensus.cc:348] T 525ce34640de48998c33bef28893ab50 P d820c59a173343aea38e08212164213d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113ef42699a14b8698690b425cfdb53a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38597 } } peers { permanent_uuid: "890cbe5f30df441dadefc35f3984b431" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44351 } } peers { permanent_uuid: "d820c59a173343aea38e08212164213d" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 36711 } }
I20260804 14:53:33.406226 4575 raft_consensus.cc:374] T 525ce34640de48998c33bef28893ab50 P d820c59a173343aea38e08212164213d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:53:33.406283 4575 raft_consensus.cc:729] T 525ce34640de48998c33bef28893ab50 P d820c59a173343aea38e08212164213d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d820c59a173343aea38e08212164213d, State: Initialized, Role: FOLLOWER
I20260804 14:53:33.406690 4574 raft_consensus.cc:348] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113ef42699a14b8698690b425cfdb53a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38597 } } peers { permanent_uuid: "890cbe5f30df441dadefc35f3984b431" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44351 } } peers { permanent_uuid: "d820c59a173343aea38e08212164213d" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 36711 } }
I20260804 14:53:33.406711 4576 raft_consensus.cc:348] T 525ce34640de48998c33bef28893ab50 P 890cbe5f30df441dadefc35f3984b431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113ef42699a14b8698690b425cfdb53a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38597 } } peers { permanent_uuid: "890cbe5f30df441dadefc35f3984b431" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44351 } } peers { permanent_uuid: "d820c59a173343aea38e08212164213d" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 36711 } }
I20260804 14:53:33.406849 4575 consensus_queue.cc:260] T 525ce34640de48998c33bef28893ab50 P d820c59a173343aea38e08212164213d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113ef42699a14b8698690b425cfdb53a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38597 } } peers { permanent_uuid: "890cbe5f30df441dadefc35f3984b431" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44351 } } peers { permanent_uuid: "d820c59a173343aea38e08212164213d" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 36711 } }
I20260804 14:53:33.406972 4574 raft_consensus.cc:374] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:53:33.406973 4576 raft_consensus.cc:374] T 525ce34640de48998c33bef28893ab50 P 890cbe5f30df441dadefc35f3984b431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:53:33.407047 4576 raft_consensus.cc:729] T 525ce34640de48998c33bef28893ab50 P 890cbe5f30df441dadefc35f3984b431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 890cbe5f30df441dadefc35f3984b431, State: Initialized, Role: FOLLOWER
I20260804 14:53:33.407043 4574 raft_consensus.cc:729] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 113ef42699a14b8698690b425cfdb53a, State: Initialized, Role: FOLLOWER
I20260804 14:53:33.407637 4149 heartbeater.cc:503] Master 127.24.6.190:35759 was elected leader, sending a full tablet report...
I20260804 14:53:33.407572 4576 consensus_queue.cc:260] T 525ce34640de48998c33bef28893ab50 P 890cbe5f30df441dadefc35f3984b431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113ef42699a14b8698690b425cfdb53a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38597 } } peers { permanent_uuid: "890cbe5f30df441dadefc35f3984b431" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44351 } } peers { permanent_uuid: "d820c59a173343aea38e08212164213d" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 36711 } }
I20260804 14:53:33.407572 4574 consensus_queue.cc:260] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113ef42699a14b8698690b425cfdb53a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38597 } } peers { permanent_uuid: "890cbe5f30df441dadefc35f3984b431" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44351 } } peers { permanent_uuid: "d820c59a173343aea38e08212164213d" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 36711 } }
I20260804 14:53:33.408537 4584 raft_consensus.cc:482] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260804 14:53:33.408677 4576 ts_tablet_manager.cc:1428] T 525ce34640de48998c33bef28893ab50 P 890cbe5f30df441dadefc35f3984b431: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260804 14:53:33.408677 4574 ts_tablet_manager.cc:1428] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260804 14:53:33.408910 4584 raft_consensus.cc:504] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113ef42699a14b8698690b425cfdb53a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38597 } } peers { permanent_uuid: "890cbe5f30df441dadefc35f3984b431" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44351 } } peers { permanent_uuid: "d820c59a173343aea38e08212164213d" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 36711 } }
I20260804 14:53:33.409070 4350 heartbeater.cc:503] Master 127.24.6.190:35759 was elected leader, sending a full tablet report...
I20260804 14:53:33.409070 4548 heartbeater.cc:503] Master 127.24.6.190:35759 was elected leader, sending a full tablet report...
I20260804 14:53:33.410177 4575 ts_tablet_manager.cc:1428] T 525ce34640de48998c33bef28893ab50 P d820c59a173343aea38e08212164213d: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260804 14:53:33.412094 4584 leader_election.cc:288] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 890cbe5f30df441dadefc35f3984b431 (127.24.6.131:44351), d820c59a173343aea38e08212164213d (127.24.6.129:36711)
I20260804 14:53:33.418210 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "113ef42699a14b8698690b425cfdb53a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: true
I20260804 14:53:33.418594 4090 raft_consensus.cc:2454] T 525ce34640de48998c33bef28893ab50 P d820c59a173343aea38e08212164213d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 113ef42699a14b8698690b425cfdb53a in term 0.
I20260804 14:53:33.418701 4481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "113ef42699a14b8698690b425cfdb53a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "890cbe5f30df441dadefc35f3984b431" is_pre_election: true
I20260804 14:53:33.419044 4481 raft_consensus.cc:2454] T 525ce34640de48998c33bef28893ab50 P 890cbe5f30df441dadefc35f3984b431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 113ef42699a14b8698690b425cfdb53a in term 0.
I20260804 14:53:33.419190 4180 leader_election.cc:302] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 113ef42699a14b8698690b425cfdb53a, d820c59a173343aea38e08212164213d; no voters:
I20260804 14:53:33.419737 4584 raft_consensus.cc:2781] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260804 14:53:33.419812 4584 raft_consensus.cc:482] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260804 14:53:33.419842 4584 raft_consensus.cc:3037] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:33.420871 4584 raft_consensus.cc:504] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113ef42699a14b8698690b425cfdb53a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38597 } } peers { permanent_uuid: "890cbe5f30df441dadefc35f3984b431" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44351 } } peers { permanent_uuid: "d820c59a173343aea38e08212164213d" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 36711 } }
I20260804 14:53:33.421298 4584 leader_election.cc:288] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a [CANDIDATE]: Term 1 election: Requested vote from peers 890cbe5f30df441dadefc35f3984b431 (127.24.6.131:44351), d820c59a173343aea38e08212164213d (127.24.6.129:36711)
I20260804 14:53:33.421780 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "113ef42699a14b8698690b425cfdb53a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d"
I20260804 14:53:33.421799 4481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "113ef42699a14b8698690b425cfdb53a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "890cbe5f30df441dadefc35f3984b431"
I20260804 14:53:33.421936 4090 raft_consensus.cc:3037] T 525ce34640de48998c33bef28893ab50 P d820c59a173343aea38e08212164213d [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:33.421949 4481 raft_consensus.cc:3037] T 525ce34640de48998c33bef28893ab50 P 890cbe5f30df441dadefc35f3984b431 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:33.423065 4090 raft_consensus.cc:2454] T 525ce34640de48998c33bef28893ab50 P d820c59a173343aea38e08212164213d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 113ef42699a14b8698690b425cfdb53a in term 1.
I20260804 14:53:33.423075 4481 raft_consensus.cc:2454] T 525ce34640de48998c33bef28893ab50 P 890cbe5f30df441dadefc35f3984b431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 113ef42699a14b8698690b425cfdb53a in term 1.
I20260804 14:53:33.423568 4176 leader_election.cc:302] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 113ef42699a14b8698690b425cfdb53a, 890cbe5f30df441dadefc35f3984b431; no voters:
I20260804 14:53:33.423911 4584 raft_consensus.cc:2781] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:33.424062 4584 raft_consensus.cc:686] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a [term 1 LEADER]: Becoming Leader. State: Replica: 113ef42699a14b8698690b425cfdb53a, State: Running, Role: LEADER
I20260804 14:53:33.424525 4584 consensus_queue.cc:237] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113ef42699a14b8698690b425cfdb53a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38597 } } peers { permanent_uuid: "890cbe5f30df441dadefc35f3984b431" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44351 } } peers { permanent_uuid: "d820c59a173343aea38e08212164213d" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 36711 } }
I20260804 14:53:33.428630 3882 catalog_manager.cc:5495] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a reported cstate change: term changed from 0 to 1, leader changed from <none> to 113ef42699a14b8698690b425cfdb53a (127.24.6.130). New cstate: current_term: 1 leader_uuid: "113ef42699a14b8698690b425cfdb53a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113ef42699a14b8698690b425cfdb53a" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "890cbe5f30df441dadefc35f3984b431" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d820c59a173343aea38e08212164213d" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 36711 } health_report { overall_health: UNKNOWN } } }
I20260804 14:53:33.448693 24602 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260804 14:53:33.450652 24602 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d820c59a173343aea38e08212164213d to finish bootstrapping
I20260804 14:53:33.458029 24602 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 113ef42699a14b8698690b425cfdb53a to finish bootstrapping
I20260804 14:53:33.465139 24602 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 890cbe5f30df441dadefc35f3984b431 to finish bootstrapping
I20260804 14:53:33.474020 4140 generic_service.cc:175] {username='slave'} at 127.0.0.1:33502 changed flags via RPC: log_inject_latency from 'false' to 'false'
I20260804 14:53:33.475695 4337 generic_service.cc:175] {username='slave'} at 127.0.0.1:32850 changed flags via RPC: log_inject_latency from 'false' to 'false'
I20260804 14:53:33.477054 4539 generic_service.cc:175] {username='slave'} at 127.0.0.1:49174 changed flags via RPC: log_inject_latency from 'false' to 'false'
I20260804 14:53:33.491005 4090 raft_consensus.cc:1261] T 525ce34640de48998c33bef28893ab50 P d820c59a173343aea38e08212164213d [term 1 FOLLOWER]: Refusing update from remote peer 113ef42699a14b8698690b425cfdb53a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260804 14:53:33.491003 4481 raft_consensus.cc:1261] T 525ce34640de48998c33bef28893ab50 P 890cbe5f30df441dadefc35f3984b431 [term 1 FOLLOWER]: Refusing update from remote peer 113ef42699a14b8698690b425cfdb53a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260804 14:53:33.491951 4585 consensus_queue.cc:1035] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a [LEADER]: Connected to new peer: Peer: permanent_uuid: "890cbe5f30df441dadefc35f3984b431" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260804 14:53:33.492169 4584 consensus_queue.cc:1035] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d820c59a173343aea38e08212164213d" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 36711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260804 14:53:33.498919 4595 mvcc.cc:205] Tried to move back new op lower bound from 7314862954451349504 to 7314862954190737408. Current Snapshot: MvccSnapshot[applied={T|T < 7314862954451349504}]
I20260804 14:53:33.499724 4596 mvcc.cc:205] Tried to move back new op lower bound from 7314862954451349504 to 7314862954190737408. Current Snapshot: MvccSnapshot[applied={T|T < 7314862954451349504}]
I20260804 14:53:33.502108 4598 mvcc.cc:205] Tried to move back new op lower bound from 7314862954451349504 to 7314862954190737408. Current Snapshot: MvccSnapshot[applied={T|T < 7314862954451349504}]
I20260804 14:53:33.517323 4337 generic_service.cc:175] {username='slave'} at 127.0.0.1:32850 changed flags via RPC: log_inject_latency_ms_mean from '100' to '1200'
I20260804 14:53:33.518316 4337 generic_service.cc:175] {username='slave'} at 127.0.0.1:32850 changed flags via RPC: log_inject_latency_ms_stddev from '100' to '0'
I20260804 14:53:33.519227 4337 generic_service.cc:175] {username='slave'} at 127.0.0.1:32850 changed flags via RPC: log_inject_latency from 'false' to 'true'
I20260804 14:53:33.520080 4140 generic_service.cc:175] {username='slave'} at 127.0.0.1:33502 changed flags via RPC: log_inject_latency_ms_mean from '100' to '600'
I20260804 14:53:33.520838 4140 generic_service.cc:175] {username='slave'} at 127.0.0.1:33502 changed flags via RPC: log_inject_latency_ms_stddev from '100' to '0'
I20260804 14:53:33.521834 4140 generic_service.cc:175] {username='slave'} at 127.0.0.1:33502 changed flags via RPC: log_inject_latency from 'false' to 'true'
I20260804 14:53:33.522380 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 4353
I20260804 14:53:33.544354 24602 test_util.cc:268] Using random seed: -778067041
I20260804 14:53:33.556576 24602 test_workload.cc:401] TestWorkload: Skipping table creation because table TestTable already exists
W20260804 14:53:33.566702 4591 log.cc:508] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a: Injecting 1200ms of latency in SegmentAllocator::Sync()
W20260804 14:53:33.567312 4176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.6.131:44351: connect: Connection refused (error 111)
W20260804 14:53:33.568560 4176 consensus_peers.cc:487] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a -> Peer 890cbe5f30df441dadefc35f3984b431 (127.24.6.131:44351): Couldn't send request to peer 890cbe5f30df441dadefc35f3984b431. Status: Network error: Client connection negotiation failed: client connection to 127.24.6.131:44351: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260804 14:53:33.568557 4601 log.cc:508] T 525ce34640de48998c33bef28893ab50 P d820c59a173343aea38e08212164213d: Injecting 600ms of latency in SegmentAllocator::Sync()
I20260804 14:53:33.608140 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 2 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:33.610344 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 2 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:33.662199 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 3 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:33.663650 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 3 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:33.715484 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 4 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:33.716980 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 4 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:33.768607 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 5 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:33.770121 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 5 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:33.821879 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 6 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:33.823733 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 6 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:33.875634 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 7 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:33.877048 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 7 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:33.928776 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 8 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:33.930332 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 8 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:33.982303 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 9 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:33.983785 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 9 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:34.035665 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 10 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:34.037120 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 10 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:34.089084 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 11 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:34.090701 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 11 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:34.142571 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 12 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:34.144227 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 12 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:34.196054 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 13 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:34.197588 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 13 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:34.249241 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 14 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:34.250742 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 14 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:34.302608 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 15 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:34.304214 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 15 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:34.356035 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 16 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:34.357468 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 16 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:34.409435 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 17 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:34.410895 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 17 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:34.462822 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 18 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:34.464591 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 18 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:34.516537 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 19 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:34.518036 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 19 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:34.569856 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 20 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:34.571442 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 20 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:34.623427 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 21 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:34.625115 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 21 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
W20260804 14:53:34.638840 4176 consensus_peers.cc:487] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a -> Peer 890cbe5f30df441dadefc35f3984b431 (127.24.6.131:44351): Couldn't send request to peer 890cbe5f30df441dadefc35f3984b431. Status: Network error: Client connection negotiation failed: client connection to 127.24.6.131:44351: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260804 14:53:34.676927 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 22 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:34.678368 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 22 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:34.730333 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 23 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:34.732661 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 23 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:34.768471 4620 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:32868 (ReqId={client: 16772c365bfc4026aeb90957a24fcdc8, seq_no=0, attempt_no=0}) took 1204 ms. Trace:
I20260804 14:53:34.768662 4620 rpcz_store.cc:276] 0804 14:53:33.564352 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:53:33.564425 (+ 73us) service_pool.cc:227] Handling call
0804 14:53:34.768443 (+1204018us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:53:33.565634 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 525ce34640de48998c33bef28893ab50
0804 14:53:33.565727 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:53:33.565732 (+ 5us) write_op.cc:435] Acquired schema lock
0804 14:53:33.565733 (+ 1us) tablet.cc:660] Decoding operations
0804 14:53:33.565804 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:53:33.565813 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0804 14:53:33.565816 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:53:33.565852 (+ 36us) tablet.cc:699] Row locks acquired
0804 14:53:33.565853 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:53:33.565974 (+ 121us) write_op.cc:270] Start()
0804 14:53:33.566001 (+ 27us) write_op.cc:276] Timestamp: P: 1785855213565892 usec, L: 0
0804 14:53:33.566002 (+ 1us) op_driver.cc:331] REPLICATION: starting
0804 14:53:33.566233 (+ 231us) log.cc:842] Serialized 345 byte log entry
0804 14:53:34.767409 (+1201176us) op_driver.cc:444] REPLICATION: finished
0804 14:53:34.767708 (+ 299us) write_op.cc:301] APPLY: starting
0804 14:53:34.767729 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0804 14:53:34.767763 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0804 14:53:34.767785 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:53:34.767881 (+ 96us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:53:34.767911 (+ 30us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:53:34.767915 (+ 4us) write_op.cc:312] APPLY: finished
0804 14:53:34.768021 (+ 106us) log.cc:842] Serialized 22 byte log entry
0804 14:53:34.768101 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:53:34.768119 (+ 18us) write_op.cc:454] Released schema lock
0804 14:53:34.768127 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":213,"num_ops":1,"prepare.queue_time_us":85,"prepare.run_cpu_time_us":761,"prepare.run_wall_time_us":761,"raft.queue_time_us":150,"raft.run_cpu_time_us":455,"raft.run_wall_time_us":455,"replication_time_us":1201362,"spinlock_wait_cycles":22016,"thread_start_us":87,"threads_started":1}]]}
I20260804 14:53:34.785071 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 24 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:34.786484 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 24 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:34.838191 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 25 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:34.839672 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 25 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:34.891376 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 26 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:34.892906 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 26 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:34.944722 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 27 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:34.946225 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 27 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
W20260804 14:53:34.973407 4591 log.cc:508] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a: Injecting 1200ms of latency in SegmentAllocator::Sync()
W20260804 14:53:34.975121 4601 log.cc:508] T 525ce34640de48998c33bef28893ab50 P d820c59a173343aea38e08212164213d: Injecting 600ms of latency in SegmentAllocator::Sync()
I20260804 14:53:34.997792 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 28 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:34.999468 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 28 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:35.051381 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 29 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:35.052801 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 29 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:35.104701 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 30 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:35.106472 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 30 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:35.158386 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 31 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:35.159806 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 31 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:35.211675 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 32 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:35.213138 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 32 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:35.264868 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 33 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:35.266492 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 33 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:35.318681 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 34 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:35.320436 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 34 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:35.372331 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 35 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:35.373974 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 35 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:35.425726 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 36 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:35.427035 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 36 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:35.478718 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 37 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:35.480232 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 37 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:35.532210 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 38 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:35.533777 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 38 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
W20260804 14:53:35.574892 4176 consensus_peers.cc:487] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a -> Peer 890cbe5f30df441dadefc35f3984b431 (127.24.6.131:44351): Couldn't send request to peer 890cbe5f30df441dadefc35f3984b431. Status: Network error: Client connection negotiation failed: client connection to 127.24.6.131:44351: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260804 14:53:35.585865 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 39 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:35.587615 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 39 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:35.639621 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 40 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:35.641665 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 40 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:35.693863 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 41 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:35.695564 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 41 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:35.747552 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 42 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:35.749053 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 42 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:35.801118 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 43 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:35.802744 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 43 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:35.854712 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 44 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:35.856427 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 44 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:35.908213 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 45 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:35.909804 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 45 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:35.961645 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 46 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:35.963299 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 46 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:36.015089 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 47 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:36.016594 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 47 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:36.068352 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 48 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:36.069835 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 48 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:36.121622 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 49 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:36.123312 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 49 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:36.174943 4627 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:32868 (ReqId={client: 16772c365bfc4026aeb90957a24fcdc8, seq_no=1, attempt_no=0}) took 1203 ms. Trace:
I20260804 14:53:36.175149 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 50 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:36.175066 4627 rpcz_store.cc:276] 0804 14:53:34.971468 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:53:34.971547 (+ 79us) service_pool.cc:227] Handling call
0804 14:53:36.174913 (+1203366us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:53:34.972539 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 525ce34640de48998c33bef28893ab50
0804 14:53:34.972643 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:53:34.972647 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:53:34.972648 (+ 1us) tablet.cc:660] Decoding operations
0804 14:53:34.972716 (+ 68us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:53:34.972749 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0804 14:53:34.972751 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:53:34.972789 (+ 38us) tablet.cc:699] Row locks acquired
0804 14:53:34.972790 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:53:34.972824 (+ 34us) write_op.cc:270] Start()
0804 14:53:34.972848 (+ 24us) write_op.cc:276] Timestamp: P: 1785855214972764 usec, L: 0
0804 14:53:34.972850 (+ 2us) op_driver.cc:331] REPLICATION: starting
0804 14:53:34.973067 (+ 217us) log.cc:842] Serialized 354 byte log entry
0804 14:53:36.173994 (+1200927us) op_driver.cc:444] REPLICATION: finished
0804 14:53:36.174320 (+ 326us) write_op.cc:301] APPLY: starting
0804 14:53:36.174345 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
0804 14:53:36.174388 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0804 14:53:36.174412 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:53:36.174489 (+ 77us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:53:36.174522 (+ 33us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:53:36.174553 (+ 31us) write_op.cc:312] APPLY: finished
0804 14:53:36.174637 (+ 84us) log.cc:842] Serialized 22 byte log entry
0804 14:53:36.174736 (+ 99us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:53:36.174757 (+ 21us) write_op.cc:454] Released schema lock
0804 14:53:36.174765 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":265,"num_ops":1,"prepare.queue_time_us":280,"prepare.run_cpu_time_us":791,"prepare.run_wall_time_us":792,"raft.queue_time_us":260,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":264,"replication_time_us":1201107,"thread_start_us":244,"threads_started":2}]]}
I20260804 14:53:36.176744 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 50 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:36.228379 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 51 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:36.229805 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 51 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:36.281612 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 52 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:36.282887 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 52 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:36.334733 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 53 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:36.336186 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 53 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
W20260804 14:53:36.379057 4591 log.cc:508] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a: Injecting 1200ms of latency in SegmentAllocator::Sync()
W20260804 14:53:36.380434 4601 log.cc:508] T 525ce34640de48998c33bef28893ab50 P d820c59a173343aea38e08212164213d: Injecting 600ms of latency in SegmentAllocator::Sync()
I20260804 14:53:36.387921 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 54 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:36.389556 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 54 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:36.441322 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 55 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:36.442895 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 55 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:36.494691 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 56 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:36.496152 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 56 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:36.548141 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 57 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:36.549641 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 57 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:36.601428 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 58 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:36.603000 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 58 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
W20260804 14:53:36.633015 4176 consensus_peers.cc:487] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a -> Peer 890cbe5f30df441dadefc35f3984b431 (127.24.6.131:44351): Couldn't send request to peer 890cbe5f30df441dadefc35f3984b431. Status: Network error: Client connection negotiation failed: client connection to 127.24.6.131:44351: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260804 14:53:36.654971 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 59 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:36.656646 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 59 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:36.708370 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 60 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:36.709869 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 60 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:36.761655 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 61 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:36.763049 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 61 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:36.815141 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 62 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:36.816635 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 62 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:36.868376 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 63 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:36.869822 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 63 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:36.921533 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 64 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:36.922977 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 64 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:36.974938 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 65 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:36.976786 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 65 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:37.028638 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 66 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:37.030161 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 66 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:37.082017 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 67 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:37.083388 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 67 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:37.135289 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 68 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:37.136812 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 68 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:37.188710 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 69 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:37.190258 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 69 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:37.242043 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 70 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:37.243688 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 70 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:37.295625 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 71 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:37.297181 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 71 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:37.349151 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 72 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:37.350605 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 72 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:37.402449 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 73 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:37.404146 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 73 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:37.456251 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 74 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:37.457729 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 74 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:37.509549 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 75 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:37.511029 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 75 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:37.563194 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 76 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:37.564987 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 76 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:37.580468 4638 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:32868 (ReqId={client: 16772c365bfc4026aeb90957a24fcdc8, seq_no=2, attempt_no=0}) took 1203 ms. Trace:
I20260804 14:53:37.580601 4638 rpcz_store.cc:276] 0804 14:53:36.377504 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:53:36.377591 (+ 87us) service_pool.cc:227] Handling call
0804 14:53:37.580440 (+1202849us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:53:36.378254 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 525ce34640de48998c33bef28893ab50
0804 14:53:36.378343 (+ 89us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:53:36.378347 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:53:36.378349 (+ 2us) tablet.cc:660] Decoding operations
0804 14:53:36.378416 (+ 67us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:53:36.378424 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0804 14:53:36.378426 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:53:36.378463 (+ 37us) tablet.cc:699] Row locks acquired
0804 14:53:36.378464 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:53:36.378498 (+ 34us) write_op.cc:270] Start()
0804 14:53:36.378524 (+ 26us) write_op.cc:276] Timestamp: P: 1785855216378442 usec, L: 0
0804 14:53:36.378526 (+ 2us) op_driver.cc:331] REPLICATION: starting
0804 14:53:36.378678 (+ 152us) log.cc:842] Serialized 354 byte log entry
0804 14:53:37.579695 (+1201017us) op_driver.cc:444] REPLICATION: finished
0804 14:53:37.579968 (+ 273us) write_op.cc:301] APPLY: starting
0804 14:53:37.579990 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0804 14:53:37.580025 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0804 14:53:37.580048 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:53:37.580125 (+ 77us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:53:37.580155 (+ 30us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:53:37.580158 (+ 3us) write_op.cc:312] APPLY: finished
0804 14:53:37.580234 (+ 76us) log.cc:842] Serialized 22 byte log entry
0804 14:53:37.580291 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:53:37.580307 (+ 16us) write_op.cc:454] Released schema lock
0804 14:53:37.580313 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":207,"num_ops":1,"prepare.queue_time_us":212,"prepare.run_cpu_time_us":633,"prepare.run_wall_time_us":634,"raft.queue_time_us":252,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":219,"replication_time_us":1201131,"thread_start_us":181,"threads_started":2}]]}
I20260804 14:53:37.616945 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 77 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:37.618606 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 77 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:37.670388 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 78 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:37.671989 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 78 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:37.723942 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 79 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:37.725634 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 79 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
W20260804 14:53:37.730525 4176 consensus_peers.cc:487] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a -> Peer 890cbe5f30df441dadefc35f3984b431 (127.24.6.131:44351): Couldn't send request to peer 890cbe5f30df441dadefc35f3984b431. Status: Network error: Client connection negotiation failed: client connection to 127.24.6.131:44351: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260804 14:53:37.777194 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 80 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:37.778633 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 80 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
W20260804 14:53:37.784050 4591 log.cc:508] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a: Injecting 1200ms of latency in SegmentAllocator::Sync()
W20260804 14:53:37.785300 4601 log.cc:508] T 525ce34640de48998c33bef28893ab50 P d820c59a173343aea38e08212164213d: Injecting 600ms of latency in SegmentAllocator::Sync()
I20260804 14:53:37.830750 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 81 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:37.832373 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 81 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:37.884253 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 82 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:37.885660 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 82 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:37.937435 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 83 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:37.938778 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 83 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:37.990562 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 84 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:37.992053 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 84 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:38.044070 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 85 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:38.045579 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 85 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:38.097251 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 86 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:38.098744 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 86 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:38.150413 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 87 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:38.151865 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 87 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:38.203629 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 88 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:38.205307 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 88 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:38.257148 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 89 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:38.258589 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 89 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:38.310423 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 90 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:38.312084 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 90 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:38.363848 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 91 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:38.365324 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 91 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:38.417061 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 92 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:38.418645 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 92 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:38.470728 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 93 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:38.472172 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 93 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:38.524030 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 94 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:38.525614 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 94 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:38.577503 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 95 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:38.579133 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 95 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:38.631013 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 96 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:38.632524 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 96 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
W20260804 14:53:38.636484 4176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.6.131:44351: connect: Connection refused (error 111) [suppressed 24 similar messages]
W20260804 14:53:38.637485 4176 consensus_peers.cc:487] T 525ce34640de48998c33bef28893ab50 P 113ef42699a14b8698690b425cfdb53a -> Peer 890cbe5f30df441dadefc35f3984b431 (127.24.6.131:44351): Couldn't send request to peer 890cbe5f30df441dadefc35f3984b431. Status: Network error: Client connection negotiation failed: client connection to 127.24.6.131:44351: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260804 14:53:38.684429 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 97 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:38.685953 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 97 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:38.737875 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 98 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:38.739605 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 98 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:38.791622 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 99 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:38.793283 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 99 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:38.845006 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 100 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:38.846369 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 100 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:38.897992 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 101 index: 0 } } ignore_live_leader: false dest_uuid: "113ef42699a14b8698690b425cfdb53a" is_pre_election: false
I20260804 14:53:38.899521 4090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ce34640de48998c33bef28893ab50" candidate_uuid: "890cbe5f30df441dadefc35f3984b431" candidate_term: 2 candidate_status { last_received { term: 101 index: 0 } } ignore_live_leader: false dest_uuid: "d820c59a173343aea38e08212164213d" is_pre_election: false
I20260804 14:53:38.985502 4648 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:32868 (ReqId={client: 16772c365bfc4026aeb90957a24fcdc8, seq_no=3, attempt_no=0}) took 1203 ms. Trace:
I20260804 14:53:38.985642 4648 rpcz_store.cc:276] 0804 14:53:37.782397 (+ 0us) service_pool.cc:170] Inserting onto call queue
0804 14:53:37.782487 (+ 90us) service_pool.cc:227] Handling call
0804 14:53:38.985477 (+1202990us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0804 14:53:37.783172 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 525ce34640de48998c33bef28893ab50
0804 14:53:37.783285 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
0804 14:53:37.783289 (+ 4us) write_op.cc:435] Acquired schema lock
0804 14:53:37.783291 (+ 2us) tablet.cc:660] Decoding operations
0804 14:53:37.783359 (+ 68us) write_op.cc:620] Acquiring the partition lock for write op
0804 14:53:37.783368 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0804 14:53:37.783370 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0804 14:53:37.783405 (+ 35us) tablet.cc:699] Row locks acquired
0804 14:53:37.783406 (+ 1us) write_op.cc:260] PREPARE: finished
0804 14:53:37.783448 (+ 42us) write_op.cc:270] Start()
0804 14:53:37.783474 (+ 26us) write_op.cc:276] Timestamp: P: 1785855217783417 usec, L: 0
0804 14:53:37.783475 (+ 1us) op_driver.cc:331] REPLICATION: starting
0804 14:53:37.783640 (+ 165us) log.cc:842] Serialized 354 byte log entry
0804 14:53:38.984666 (+1201026us) op_driver.cc:444] REPLICATION: finished
0804 14:53:38.984994 (+ 328us) write_op.cc:301] APPLY: starting
0804 14:53:38.985015 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0804 14:53:38.985048 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0804 14:53:38.985070 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0804 14:53:38.985142 (+ 72us) tablet.cc:1381] finished ApplyRowOperation cycle
0804 14:53:38.985172 (+ 30us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0804 14:53:38.985175 (+ 3us) write_op.cc:312] APPLY: finished
0804 14:53:38.985287 (+ 112us) log.cc:842] Serialized 22 byte log entry
0804 14:53:38.985360 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0804 14:53:38.985377 (+ 17us) write_op.cc:454] Released schema lock
0804 14:53:38.985384 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":254,"num_ops":1,"prepare.queue_time_us":230,"prepare.run_cpu_time_us":625,"prepare.run_wall_time_us":625,"raft.queue_time_us":161,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":185,"replication_time_us":1201155,"thread_start_us":201,"threads_started":2}]]}
I20260804 14:53:39.187530 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 3970
I20260804 14:53:39.200453 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 4153
I20260804 14:53:39.216559 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 3834
2026-08-04T14:53:39Z chronyd exiting
[ OK ] RaftConsensusElectionITest.DisruptiveServerAndSlowWAL (6875 ms)
[----------] 12 tests from RaftConsensusElectionITest (171112 ms total)
[----------] 2 tests from NumReplicas/RaftConsensusNumLeadersMetricTest
[ RUN ] NumReplicas/RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric/0
I20260804 14:53:39.249176 24602 test_util.cc:268] Using random seed: -772362224
I20260804 14:53:39.250205 24602 ts_itest-base.cc:112] Starting cluster with:
I20260804 14:53:39.250275 24602 ts_itest-base.cc:113] --------------
I20260804 14:53:39.250298 24602 ts_itest-base.cc:114] 1 tablet servers
I20260804 14:53:39.250326 24602 ts_itest-base.cc:115] 3 replicas per TS
I20260804 14:53:39.250344 24602 ts_itest-base.cc:116] --------------
2026-08-04T14:53:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-04T14:53:39Z Disabled control of system clock
I20260804 14:53:39.266369 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.6.190:42761
--webserver_interface=127.24.6.190
--webserver_port=0
--builtin_ntp_servers=127.24.6.148:37609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.6.190:42761
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20260804 14:53:39.393594 4668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:39.393931 4668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:39.393975 4668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:39.398263 4668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260804 14:53:39.398350 4668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:39.398368 4668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260804 14:53:39.398386 4668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260804 14:53:39.403537 4668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:37609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.24.6.190:42761
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.6.190:42761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.24.6.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.4668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:39.404747 4668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:39.406188 4668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:39.415916 4678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:39.415916 4674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:39.416064 4675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:39.416114 4668 server_base.cc:1034] running on GCE node
I20260804 14:53:39.416960 4668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:39.418365 4668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:39.419623 4668 hybrid_clock.cc:648] HybridClock initialized: now 1785855219419598 us; error 40 us; skew 500 ppm
I20260804 14:53:39.427896 4668 webserver.cc:460] Webserver started at http://127.24.6.190:40247/ using document root <none> and password file <none>
I20260804 14:53:39.428649 4668 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:39.428758 4668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:39.429020 4668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:39.431298 4668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "0093666d0f274b1f850313b8ef5df955"
format_stamp: "Formatted at 2026-08-04 14:53:39 on dist-test-slave-92m1"
I20260804 14:53:39.431955 4668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "0093666d0f274b1f850313b8ef5df955"
format_stamp: "Formatted at 2026-08-04 14:53:39 on dist-test-slave-92m1"
I20260804 14:53:39.436445 4668 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260804 14:53:39.439558 4684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:39.441210 4668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260804 14:53:39.441465 4668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
uuid: "0093666d0f274b1f850313b8ef5df955"
format_stamp: "Formatted at 2026-08-04 14:53:39 on dist-test-slave-92m1"
I20260804 14:53:39.441627 4668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:39.452859 4668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:39.453703 4668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:39.453920 4668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:39.461669 4668 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.190:42761
I20260804 14:53:39.461683 4759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.190:42761 every 8 connection(s)
I20260804 14:53:39.462759 4668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260804 14:53:39.465687 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 4668
I20260804 14:53:39.465827 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260804 14:53:39.466842 4760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:39.473603 4760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0093666d0f274b1f850313b8ef5df955: Bootstrap starting.
I20260804 14:53:39.476815 4760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0093666d0f274b1f850313b8ef5df955: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:39.477713 4760 log.cc:824] T 00000000000000000000000000000000 P 0093666d0f274b1f850313b8ef5df955: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:39.480154 4760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0093666d0f274b1f850313b8ef5df955: No bootstrap required, opened a new log
I20260804 14:53:39.484494 4760 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0093666d0f274b1f850313b8ef5df955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0093666d0f274b1f850313b8ef5df955" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 42761 } }
I20260804 14:53:39.484794 4760 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0093666d0f274b1f850313b8ef5df955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:53:39.484870 4760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0093666d0f274b1f850313b8ef5df955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0093666d0f274b1f850313b8ef5df955, State: Initialized, Role: FOLLOWER
I20260804 14:53:39.485394 4760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0093666d0f274b1f850313b8ef5df955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0093666d0f274b1f850313b8ef5df955" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 42761 } }
I20260804 14:53:39.485549 4760 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0093666d0f274b1f850313b8ef5df955 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260804 14:53:39.485646 4760 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0093666d0f274b1f850313b8ef5df955 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260804 14:53:39.485759 4760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0093666d0f274b1f850313b8ef5df955 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:39.486779 4760 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0093666d0f274b1f850313b8ef5df955 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0093666d0f274b1f850313b8ef5df955" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 42761 } }
I20260804 14:53:39.487185 4760 leader_election.cc:302] T 00000000000000000000000000000000 P 0093666d0f274b1f850313b8ef5df955 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0093666d0f274b1f850313b8ef5df955; no voters:
I20260804 14:53:39.487623 4760 leader_election.cc:288] T 00000000000000000000000000000000 P 0093666d0f274b1f850313b8ef5df955 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260804 14:53:39.487769 4769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0093666d0f274b1f850313b8ef5df955 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:39.488058 4769 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0093666d0f274b1f850313b8ef5df955 [term 1 LEADER]: Becoming Leader. State: Replica: 0093666d0f274b1f850313b8ef5df955, State: Running, Role: LEADER
I20260804 14:53:39.488426 4769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0093666d0f274b1f850313b8ef5df955 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0093666d0f274b1f850313b8ef5df955" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 42761 } }
I20260804 14:53:39.489122 4760 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0093666d0f274b1f850313b8ef5df955 [sys.catalog]: configured and running, proceeding with master startup.
I20260804 14:53:39.490423 4770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0093666d0f274b1f850313b8ef5df955 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0093666d0f274b1f850313b8ef5df955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0093666d0f274b1f850313b8ef5df955" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 42761 } } }
I20260804 14:53:39.490578 4770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0093666d0f274b1f850313b8ef5df955 [sys.catalog]: This master's current role is: LEADER
I20260804 14:53:39.490899 4771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0093666d0f274b1f850313b8ef5df955 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0093666d0f274b1f850313b8ef5df955. Latest consensus state: current_term: 1 leader_uuid: "0093666d0f274b1f850313b8ef5df955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0093666d0f274b1f850313b8ef5df955" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 42761 } } }
I20260804 14:53:39.491040 4771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0093666d0f274b1f850313b8ef5df955 [sys.catalog]: This master's current role is: LEADER
I20260804 14:53:39.491132 4779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260804 14:53:39.494863 4779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260804 14:53:39.501600 4779 catalog_manager.cc:1347] Generated new cluster ID: c7f436bd233e47b698062b9c86045717
I20260804 14:53:39.501713 4779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260804 14:53:39.522744 4779 catalog_manager.cc:1370] Generated new certificate authority record
I20260804 14:53:39.523924 4779 catalog_manager.cc:1504] Loading token signing keys...
I20260804 14:53:39.533768 4779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0093666d0f274b1f850313b8ef5df955: Generated new TSK 0
I20260804 14:53:39.534768 4779 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260804 14:53:39.548870 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.129:0
--local_ip_for_outbound_sockets=127.24.6.129
--webserver_interface=127.24.6.129
--webserver_port=0
--tserver_master_addrs=127.24.6.190:42761
--builtin_ntp_servers=127.24.6.148:37609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20260804 14:53:39.681635 4791 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260804 14:53:39.681983 4791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:39.682006 4791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:39.682041 4791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:39.686223 4791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:39.686340 4791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.129
I20260804 14:53:39.692093 4791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:37609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.24.6.129
--webserver_port=0
--tserver_master_addrs=127.24.6.190:42761
--never_fsync=true
--heap_profile_path=/tmp/kudu.4791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.129
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:39.693563 4791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:39.695151 4791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:39.705919 4802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:39.705884 4799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:39.706036 4797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:39.706197 4791 server_base.cc:1034] running on GCE node
I20260804 14:53:39.707069 4791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:39.707839 4791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:39.709038 4791 hybrid_clock.cc:648] HybridClock initialized: now 1785855219709014 us; error 42 us; skew 500 ppm
I20260804 14:53:39.711382 4791 webserver.cc:460] Webserver started at http://127.24.6.129:42023/ using document root <none> and password file <none>
I20260804 14:53:39.712052 4791 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:39.712126 4791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:39.712328 4791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:39.714546 4791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "88e20b82c7674c67a918e579365cd514"
format_stamp: "Formatted at 2026-08-04 14:53:39 on dist-test-slave-92m1"
I20260804 14:53:39.715167 4791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "88e20b82c7674c67a918e579365cd514"
format_stamp: "Formatted at 2026-08-04 14:53:39 on dist-test-slave-92m1"
I20260804 14:53:39.719529 4791 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260804 14:53:39.722222 4812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:39.723734 4791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260804 14:53:39.723888 4791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "88e20b82c7674c67a918e579365cd514"
format_stamp: "Formatted at 2026-08-04 14:53:39 on dist-test-slave-92m1"
I20260804 14:53:39.724030 4791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:39.744275 4791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:39.745100 4791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:39.745314 4791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:39.746033 4791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:53:39.747202 4791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:53:39.747301 4791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:39.747371 4791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:53:39.747416 4791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:39.757231 4791 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.129:41155
I20260804 14:53:39.757251 4971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.129:41155 every 8 connection(s)
I20260804 14:53:39.758437 4791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260804 14:53:39.768806 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 4791
I20260804 14:53:39.769026 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260804 14:53:39.770548 4972 heartbeater.cc:347] Connected to a master server at 127.24.6.190:42761
I20260804 14:53:39.770836 4972 heartbeater.cc:464] Registering TS with master...
I20260804 14:53:39.771548 4972 heartbeater.cc:511] Master 127.24.6.190:42761 requested a full tablet report, sending...
I20260804 14:53:39.773291 4704 ts_manager.cc:194] Registered new tserver with Master: 88e20b82c7674c67a918e579365cd514 (127.24.6.129:41155)
I20260804 14:53:39.774673 4704 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.129:41705
I20260804 14:53:39.783959 24602 external_mini_cluster.cc:930] 1 TS(s) registered with all masters
I20260804 14:53:39.795663 24602 external_mini_cluster.cc:930] 1 TS(s) registered with all masters
I20260804 14:53:39.795749 24602 test_util.cc:268] Using random seed: -771815642
I20260804 14:53:39.807942 4704 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32958:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260804 14:53:39.831001 4859 tablet_service.cc:1467] Processing CreateTablet for tablet cca9b26f4ab5407ebc2ba2c2ea82953e (DEFAULT_TABLE table=test-workload [id=e924b8a5cb11400087548239730ffd72]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260804 14:53:39.830154 4873 tablet_service.cc:1467] Processing CreateTablet for tablet 33d23dc00de74072bccf05a1afa9b62a (DEFAULT_TABLE table=test-workload [id=e924b8a5cb11400087548239730ffd72]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260804 14:53:39.830874 4863 tablet_service.cc:1467] Processing CreateTablet for tablet 5a47a19b1204470ba09da1170ea89400 (DEFAULT_TABLE table=test-workload [id=e924b8a5cb11400087548239730ffd72]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260804 14:53:39.831307 4862 tablet_service.cc:1467] Processing CreateTablet for tablet 2f36d606de4647b695d8b82f70e1f012 (DEFAULT_TABLE table=test-workload [id=e924b8a5cb11400087548239730ffd72]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260804 14:53:39.830154 4872 tablet_service.cc:1467] Processing CreateTablet for tablet 06d5a3e284c04301b765ec23671f0eee (DEFAULT_TABLE table=test-workload [id=e924b8a5cb11400087548239730ffd72]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260804 14:53:39.830155 4875 tablet_service.cc:1467] Processing CreateTablet for tablet a1c71c43dcba46b59519f16d317472fe (DEFAULT_TABLE table=test-workload [id=e924b8a5cb11400087548239730ffd72]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260804 14:53:39.830754 4870 tablet_service.cc:1467] Processing CreateTablet for tablet 530fa576e66047d3b2f0838894b7638c (DEFAULT_TABLE table=test-workload [id=e924b8a5cb11400087548239730ffd72]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260804 14:53:39.830276 4869 tablet_service.cc:1467] Processing CreateTablet for tablet 37f31df0068c4c4e921cc713475fd55e (DEFAULT_TABLE table=test-workload [id=e924b8a5cb11400087548239730ffd72]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260804 14:53:39.830359 4868 tablet_service.cc:1467] Processing CreateTablet for tablet 17189dbd7cfa4ca9a4b7aecf77e689ff (DEFAULT_TABLE table=test-workload [id=e924b8a5cb11400087548239730ffd72]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260804 14:53:39.830155 4877 tablet_service.cc:1467] Processing CreateTablet for tablet a941152217e240ffa03ac2c950271763 (DEFAULT_TABLE table=test-workload [id=e924b8a5cb11400087548239730ffd72]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260804 14:53:39.832208 4875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1c71c43dcba46b59519f16d317472fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:39.832641 4870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 530fa576e66047d3b2f0838894b7638c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:39.832805 4862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f36d606de4647b695d8b82f70e1f012. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:39.833688 4869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37f31df0068c4c4e921cc713475fd55e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:39.833864 4863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a47a19b1204470ba09da1170ea89400. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:39.834160 4868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17189dbd7cfa4ca9a4b7aecf77e689ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:39.834496 4859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cca9b26f4ab5407ebc2ba2c2ea82953e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:39.834969 4872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06d5a3e284c04301b765ec23671f0eee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:39.839383 4877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a941152217e240ffa03ac2c950271763. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:39.839598 4873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33d23dc00de74072bccf05a1afa9b62a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:39.843577 4993 tablet_bootstrap.cc:492] T 5a47a19b1204470ba09da1170ea89400 P 88e20b82c7674c67a918e579365cd514: Bootstrap starting.
I20260804 14:53:39.846619 4993 tablet_bootstrap.cc:654] T 5a47a19b1204470ba09da1170ea89400 P 88e20b82c7674c67a918e579365cd514: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:39.847922 4993 log.cc:824] T 5a47a19b1204470ba09da1170ea89400 P 88e20b82c7674c67a918e579365cd514: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:39.850394 4993 tablet_bootstrap.cc:492] T 5a47a19b1204470ba09da1170ea89400 P 88e20b82c7674c67a918e579365cd514: No bootstrap required, opened a new log
I20260804 14:53:39.850718 4993 ts_tablet_manager.cc:1397] T 5a47a19b1204470ba09da1170ea89400 P 88e20b82c7674c67a918e579365cd514: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260804 14:53:39.854357 4993 raft_consensus.cc:348] T 5a47a19b1204470ba09da1170ea89400 P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:39.854645 4993 raft_consensus.cc:729] T 5a47a19b1204470ba09da1170ea89400 P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88e20b82c7674c67a918e579365cd514, State: Initialized, Role: FOLLOWER
I20260804 14:53:39.855039 4993 consensus_queue.cc:260] T 5a47a19b1204470ba09da1170ea89400 P 88e20b82c7674c67a918e579365cd514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:39.855985 4972 heartbeater.cc:503] Master 127.24.6.190:42761 was elected leader, sending a full tablet report...
I20260804 14:53:39.856251 4993 ts_tablet_manager.cc:1428] T 5a47a19b1204470ba09da1170ea89400 P 88e20b82c7674c67a918e579365cd514: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260804 14:53:39.856565 4993 tablet_bootstrap.cc:492] T 06d5a3e284c04301b765ec23671f0eee P 88e20b82c7674c67a918e579365cd514: Bootstrap starting.
I20260804 14:53:39.857791 4993 tablet_bootstrap.cc:654] T 06d5a3e284c04301b765ec23671f0eee P 88e20b82c7674c67a918e579365cd514: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:39.859517 4993 tablet_bootstrap.cc:492] T 06d5a3e284c04301b765ec23671f0eee P 88e20b82c7674c67a918e579365cd514: No bootstrap required, opened a new log
I20260804 14:53:39.859652 4993 ts_tablet_manager.cc:1397] T 06d5a3e284c04301b765ec23671f0eee P 88e20b82c7674c67a918e579365cd514: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260804 14:53:39.860479 4993 raft_consensus.cc:348] T 06d5a3e284c04301b765ec23671f0eee P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:39.860611 4993 raft_consensus.cc:729] T 06d5a3e284c04301b765ec23671f0eee P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88e20b82c7674c67a918e579365cd514, State: Initialized, Role: FOLLOWER
I20260804 14:53:39.860719 4993 consensus_queue.cc:260] T 06d5a3e284c04301b765ec23671f0eee P 88e20b82c7674c67a918e579365cd514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:39.861042 4993 ts_tablet_manager.cc:1428] T 06d5a3e284c04301b765ec23671f0eee P 88e20b82c7674c67a918e579365cd514: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260804 14:53:39.861264 4993 tablet_bootstrap.cc:492] T cca9b26f4ab5407ebc2ba2c2ea82953e P 88e20b82c7674c67a918e579365cd514: Bootstrap starting.
I20260804 14:53:39.862340 4993 tablet_bootstrap.cc:654] T cca9b26f4ab5407ebc2ba2c2ea82953e P 88e20b82c7674c67a918e579365cd514: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:39.864392 4993 tablet_bootstrap.cc:492] T cca9b26f4ab5407ebc2ba2c2ea82953e P 88e20b82c7674c67a918e579365cd514: No bootstrap required, opened a new log
I20260804 14:53:39.864508 4993 ts_tablet_manager.cc:1397] T cca9b26f4ab5407ebc2ba2c2ea82953e P 88e20b82c7674c67a918e579365cd514: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260804 14:53:39.864992 4993 raft_consensus.cc:348] T cca9b26f4ab5407ebc2ba2c2ea82953e P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:39.865131 4993 raft_consensus.cc:729] T cca9b26f4ab5407ebc2ba2c2ea82953e P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88e20b82c7674c67a918e579365cd514, State: Initialized, Role: FOLLOWER
I20260804 14:53:39.865229 4993 consensus_queue.cc:260] T cca9b26f4ab5407ebc2ba2c2ea82953e P 88e20b82c7674c67a918e579365cd514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:39.865543 4993 ts_tablet_manager.cc:1428] T cca9b26f4ab5407ebc2ba2c2ea82953e P 88e20b82c7674c67a918e579365cd514: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260804 14:53:39.865746 4993 tablet_bootstrap.cc:492] T 2f36d606de4647b695d8b82f70e1f012 P 88e20b82c7674c67a918e579365cd514: Bootstrap starting.
I20260804 14:53:39.866801 4993 tablet_bootstrap.cc:654] T 2f36d606de4647b695d8b82f70e1f012 P 88e20b82c7674c67a918e579365cd514: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:39.868171 4993 tablet_bootstrap.cc:492] T 2f36d606de4647b695d8b82f70e1f012 P 88e20b82c7674c67a918e579365cd514: No bootstrap required, opened a new log
I20260804 14:53:39.868285 4993 ts_tablet_manager.cc:1397] T 2f36d606de4647b695d8b82f70e1f012 P 88e20b82c7674c67a918e579365cd514: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260804 14:53:39.868734 4993 raft_consensus.cc:348] T 2f36d606de4647b695d8b82f70e1f012 P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:39.868851 4993 raft_consensus.cc:729] T 2f36d606de4647b695d8b82f70e1f012 P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88e20b82c7674c67a918e579365cd514, State: Initialized, Role: FOLLOWER
I20260804 14:53:39.868961 4993 consensus_queue.cc:260] T 2f36d606de4647b695d8b82f70e1f012 P 88e20b82c7674c67a918e579365cd514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:39.869256 4993 ts_tablet_manager.cc:1428] T 2f36d606de4647b695d8b82f70e1f012 P 88e20b82c7674c67a918e579365cd514: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260804 14:53:39.869473 4993 tablet_bootstrap.cc:492] T a1c71c43dcba46b59519f16d317472fe P 88e20b82c7674c67a918e579365cd514: Bootstrap starting.
I20260804 14:53:39.870465 4993 tablet_bootstrap.cc:654] T a1c71c43dcba46b59519f16d317472fe P 88e20b82c7674c67a918e579365cd514: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:39.871845 4993 tablet_bootstrap.cc:492] T a1c71c43dcba46b59519f16d317472fe P 88e20b82c7674c67a918e579365cd514: No bootstrap required, opened a new log
I20260804 14:53:39.871960 4993 ts_tablet_manager.cc:1397] T a1c71c43dcba46b59519f16d317472fe P 88e20b82c7674c67a918e579365cd514: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260804 14:53:39.872370 4993 raft_consensus.cc:348] T a1c71c43dcba46b59519f16d317472fe P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:39.872498 4993 raft_consensus.cc:729] T a1c71c43dcba46b59519f16d317472fe P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88e20b82c7674c67a918e579365cd514, State: Initialized, Role: FOLLOWER
I20260804 14:53:39.872612 4993 consensus_queue.cc:260] T a1c71c43dcba46b59519f16d317472fe P 88e20b82c7674c67a918e579365cd514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:39.872962 4993 ts_tablet_manager.cc:1428] T a1c71c43dcba46b59519f16d317472fe P 88e20b82c7674c67a918e579365cd514: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260804 14:53:39.873235 4993 tablet_bootstrap.cc:492] T a941152217e240ffa03ac2c950271763 P 88e20b82c7674c67a918e579365cd514: Bootstrap starting.
I20260804 14:53:39.874181 4993 tablet_bootstrap.cc:654] T a941152217e240ffa03ac2c950271763 P 88e20b82c7674c67a918e579365cd514: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:39.875691 4993 tablet_bootstrap.cc:492] T a941152217e240ffa03ac2c950271763 P 88e20b82c7674c67a918e579365cd514: No bootstrap required, opened a new log
I20260804 14:53:39.875787 4993 ts_tablet_manager.cc:1397] T a941152217e240ffa03ac2c950271763 P 88e20b82c7674c67a918e579365cd514: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260804 14:53:39.876398 4993 raft_consensus.cc:348] T a941152217e240ffa03ac2c950271763 P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:39.876539 4993 raft_consensus.cc:729] T a941152217e240ffa03ac2c950271763 P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88e20b82c7674c67a918e579365cd514, State: Initialized, Role: FOLLOWER
I20260804 14:53:39.876654 4993 consensus_queue.cc:260] T a941152217e240ffa03ac2c950271763 P 88e20b82c7674c67a918e579365cd514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:39.876946 4993 ts_tablet_manager.cc:1428] T a941152217e240ffa03ac2c950271763 P 88e20b82c7674c67a918e579365cd514: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260804 14:53:39.877084 4993 tablet_bootstrap.cc:492] T 530fa576e66047d3b2f0838894b7638c P 88e20b82c7674c67a918e579365cd514: Bootstrap starting.
I20260804 14:53:39.878111 4993 tablet_bootstrap.cc:654] T 530fa576e66047d3b2f0838894b7638c P 88e20b82c7674c67a918e579365cd514: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:39.879498 4993 tablet_bootstrap.cc:492] T 530fa576e66047d3b2f0838894b7638c P 88e20b82c7674c67a918e579365cd514: No bootstrap required, opened a new log
I20260804 14:53:39.879586 4993 ts_tablet_manager.cc:1397] T 530fa576e66047d3b2f0838894b7638c P 88e20b82c7674c67a918e579365cd514: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260804 14:53:39.880086 4993 raft_consensus.cc:348] T 530fa576e66047d3b2f0838894b7638c P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:39.880182 4993 raft_consensus.cc:729] T 530fa576e66047d3b2f0838894b7638c P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88e20b82c7674c67a918e579365cd514, State: Initialized, Role: FOLLOWER
I20260804 14:53:39.880292 4993 consensus_queue.cc:260] T 530fa576e66047d3b2f0838894b7638c P 88e20b82c7674c67a918e579365cd514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:39.880668 4993 ts_tablet_manager.cc:1428] T 530fa576e66047d3b2f0838894b7638c P 88e20b82c7674c67a918e579365cd514: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260804 14:53:39.880877 4993 tablet_bootstrap.cc:492] T 37f31df0068c4c4e921cc713475fd55e P 88e20b82c7674c67a918e579365cd514: Bootstrap starting.
I20260804 14:53:39.881750 4993 tablet_bootstrap.cc:654] T 37f31df0068c4c4e921cc713475fd55e P 88e20b82c7674c67a918e579365cd514: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:39.883271 4993 tablet_bootstrap.cc:492] T 37f31df0068c4c4e921cc713475fd55e P 88e20b82c7674c67a918e579365cd514: No bootstrap required, opened a new log
I20260804 14:53:39.883383 4993 ts_tablet_manager.cc:1397] T 37f31df0068c4c4e921cc713475fd55e P 88e20b82c7674c67a918e579365cd514: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260804 14:53:39.883812 4993 raft_consensus.cc:348] T 37f31df0068c4c4e921cc713475fd55e P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:39.883924 4993 raft_consensus.cc:729] T 37f31df0068c4c4e921cc713475fd55e P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88e20b82c7674c67a918e579365cd514, State: Initialized, Role: FOLLOWER
I20260804 14:53:39.884049 4993 consensus_queue.cc:260] T 37f31df0068c4c4e921cc713475fd55e P 88e20b82c7674c67a918e579365cd514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:39.884315 4993 ts_tablet_manager.cc:1428] T 37f31df0068c4c4e921cc713475fd55e P 88e20b82c7674c67a918e579365cd514: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260804 14:53:39.884507 4993 tablet_bootstrap.cc:492] T 17189dbd7cfa4ca9a4b7aecf77e689ff P 88e20b82c7674c67a918e579365cd514: Bootstrap starting.
I20260804 14:53:39.885426 4993 tablet_bootstrap.cc:654] T 17189dbd7cfa4ca9a4b7aecf77e689ff P 88e20b82c7674c67a918e579365cd514: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:39.887025 4993 tablet_bootstrap.cc:492] T 17189dbd7cfa4ca9a4b7aecf77e689ff P 88e20b82c7674c67a918e579365cd514: No bootstrap required, opened a new log
I20260804 14:53:39.887164 4993 ts_tablet_manager.cc:1397] T 17189dbd7cfa4ca9a4b7aecf77e689ff P 88e20b82c7674c67a918e579365cd514: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260804 14:53:39.887796 4993 raft_consensus.cc:348] T 17189dbd7cfa4ca9a4b7aecf77e689ff P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:39.887892 4993 raft_consensus.cc:729] T 17189dbd7cfa4ca9a4b7aecf77e689ff P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88e20b82c7674c67a918e579365cd514, State: Initialized, Role: FOLLOWER
I20260804 14:53:39.887979 4993 consensus_queue.cc:260] T 17189dbd7cfa4ca9a4b7aecf77e689ff P 88e20b82c7674c67a918e579365cd514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:39.888266 4993 ts_tablet_manager.cc:1428] T 17189dbd7cfa4ca9a4b7aecf77e689ff P 88e20b82c7674c67a918e579365cd514: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260804 14:53:39.888485 4993 tablet_bootstrap.cc:492] T 33d23dc00de74072bccf05a1afa9b62a P 88e20b82c7674c67a918e579365cd514: Bootstrap starting.
I20260804 14:53:39.889500 4993 tablet_bootstrap.cc:654] T 33d23dc00de74072bccf05a1afa9b62a P 88e20b82c7674c67a918e579365cd514: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:39.891129 4993 tablet_bootstrap.cc:492] T 33d23dc00de74072bccf05a1afa9b62a P 88e20b82c7674c67a918e579365cd514: No bootstrap required, opened a new log
I20260804 14:53:39.891256 4993 ts_tablet_manager.cc:1397] T 33d23dc00de74072bccf05a1afa9b62a P 88e20b82c7674c67a918e579365cd514: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260804 14:53:39.891770 4993 raft_consensus.cc:348] T 33d23dc00de74072bccf05a1afa9b62a P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:39.891883 4993 raft_consensus.cc:729] T 33d23dc00de74072bccf05a1afa9b62a P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88e20b82c7674c67a918e579365cd514, State: Initialized, Role: FOLLOWER
I20260804 14:53:39.891986 4993 consensus_queue.cc:260] T 33d23dc00de74072bccf05a1afa9b62a P 88e20b82c7674c67a918e579365cd514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:39.892274 4993 ts_tablet_manager.cc:1428] T 33d23dc00de74072bccf05a1afa9b62a P 88e20b82c7674c67a918e579365cd514: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260804 14:53:39.931921 4904 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "33d23dc00de74072bccf05a1afa9b62a"
dest_uuid: "88e20b82c7674c67a918e579365cd514"
from {username='slave'} at 127.0.0.1:51762
I20260804 14:53:39.932240 4904 raft_consensus.cc:482] T 33d23dc00de74072bccf05a1afa9b62a P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260804 14:53:39.932404 4904 raft_consensus.cc:3037] T 33d23dc00de74072bccf05a1afa9b62a P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:39.933588 4904 raft_consensus.cc:504] T 33d23dc00de74072bccf05a1afa9b62a P 88e20b82c7674c67a918e579365cd514 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:39.934024 4904 leader_election.cc:302] T 33d23dc00de74072bccf05a1afa9b62a P 88e20b82c7674c67a918e579365cd514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88e20b82c7674c67a918e579365cd514; no voters:
I20260804 14:53:39.934301 4904 leader_election.cc:288] T 33d23dc00de74072bccf05a1afa9b62a P 88e20b82c7674c67a918e579365cd514 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260804 14:53:39.934413 4997 raft_consensus.cc:2781] T 33d23dc00de74072bccf05a1afa9b62a P 88e20b82c7674c67a918e579365cd514 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:39.934741 4997 raft_consensus.cc:686] T 33d23dc00de74072bccf05a1afa9b62a P 88e20b82c7674c67a918e579365cd514 [term 1 LEADER]: Becoming Leader. State: Replica: 88e20b82c7674c67a918e579365cd514, State: Running, Role: LEADER
I20260804 14:53:39.935156 4997 consensus_queue.cc:237] T 33d23dc00de74072bccf05a1afa9b62a P 88e20b82c7674c67a918e579365cd514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:39.938563 4702 catalog_manager.cc:5495] T 33d23dc00de74072bccf05a1afa9b62a P 88e20b82c7674c67a918e579365cd514 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88e20b82c7674c67a918e579365cd514 (127.24.6.129). New cstate: current_term: 1 leader_uuid: "88e20b82c7674c67a918e579365cd514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } health_report { overall_health: HEALTHY } } }
I20260804 14:53:39.944511 4904 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "530fa576e66047d3b2f0838894b7638c"
dest_uuid: "88e20b82c7674c67a918e579365cd514"
from {username='slave'} at 127.0.0.1:51762
I20260804 14:53:39.944736 4904 raft_consensus.cc:482] T 530fa576e66047d3b2f0838894b7638c P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260804 14:53:39.944810 4904 raft_consensus.cc:3037] T 530fa576e66047d3b2f0838894b7638c P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:39.945912 4904 raft_consensus.cc:504] T 530fa576e66047d3b2f0838894b7638c P 88e20b82c7674c67a918e579365cd514 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:39.946090 4904 leader_election.cc:302] T 530fa576e66047d3b2f0838894b7638c P 88e20b82c7674c67a918e579365cd514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88e20b82c7674c67a918e579365cd514; no voters:
I20260804 14:53:39.946223 4904 leader_election.cc:288] T 530fa576e66047d3b2f0838894b7638c P 88e20b82c7674c67a918e579365cd514 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260804 14:53:39.946270 4997 raft_consensus.cc:2781] T 530fa576e66047d3b2f0838894b7638c P 88e20b82c7674c67a918e579365cd514 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:39.946360 4997 raft_consensus.cc:686] T 530fa576e66047d3b2f0838894b7638c P 88e20b82c7674c67a918e579365cd514 [term 1 LEADER]: Becoming Leader. State: Replica: 88e20b82c7674c67a918e579365cd514, State: Running, Role: LEADER
I20260804 14:53:39.946557 4997 consensus_queue.cc:237] T 530fa576e66047d3b2f0838894b7638c P 88e20b82c7674c67a918e579365cd514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:39.948451 4702 catalog_manager.cc:5495] T 530fa576e66047d3b2f0838894b7638c P 88e20b82c7674c67a918e579365cd514 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88e20b82c7674c67a918e579365cd514 (127.24.6.129). New cstate: current_term: 1 leader_uuid: "88e20b82c7674c67a918e579365cd514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } health_report { overall_health: HEALTHY } } }
I20260804 14:53:39.955541 4904 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "37f31df0068c4c4e921cc713475fd55e"
dest_uuid: "88e20b82c7674c67a918e579365cd514"
from {username='slave'} at 127.0.0.1:51762
I20260804 14:53:39.955725 4904 raft_consensus.cc:482] T 37f31df0068c4c4e921cc713475fd55e P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260804 14:53:39.955782 4904 raft_consensus.cc:3037] T 37f31df0068c4c4e921cc713475fd55e P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:39.956894 4904 raft_consensus.cc:504] T 37f31df0068c4c4e921cc713475fd55e P 88e20b82c7674c67a918e579365cd514 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:39.957060 4904 leader_election.cc:302] T 37f31df0068c4c4e921cc713475fd55e P 88e20b82c7674c67a918e579365cd514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88e20b82c7674c67a918e579365cd514; no voters:
I20260804 14:53:39.957185 4904 leader_election.cc:288] T 37f31df0068c4c4e921cc713475fd55e P 88e20b82c7674c67a918e579365cd514 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260804 14:53:39.957237 4997 raft_consensus.cc:2781] T 37f31df0068c4c4e921cc713475fd55e P 88e20b82c7674c67a918e579365cd514 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:39.957332 4997 raft_consensus.cc:686] T 37f31df0068c4c4e921cc713475fd55e P 88e20b82c7674c67a918e579365cd514 [term 1 LEADER]: Becoming Leader. State: Replica: 88e20b82c7674c67a918e579365cd514, State: Running, Role: LEADER
I20260804 14:53:39.957525 4997 consensus_queue.cc:237] T 37f31df0068c4c4e921cc713475fd55e P 88e20b82c7674c67a918e579365cd514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:39.959340 4702 catalog_manager.cc:5495] T 37f31df0068c4c4e921cc713475fd55e P 88e20b82c7674c67a918e579365cd514 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88e20b82c7674c67a918e579365cd514 (127.24.6.129). New cstate: current_term: 1 leader_uuid: "88e20b82c7674c67a918e579365cd514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } health_report { overall_health: HEALTHY } } }
I20260804 14:53:39.967123 4904 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "06d5a3e284c04301b765ec23671f0eee"
dest_uuid: "88e20b82c7674c67a918e579365cd514"
from {username='slave'} at 127.0.0.1:51762
I20260804 14:53:39.967805 4904 raft_consensus.cc:482] T 06d5a3e284c04301b765ec23671f0eee P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260804 14:53:39.967896 4904 raft_consensus.cc:3037] T 06d5a3e284c04301b765ec23671f0eee P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:39.969712 4904 raft_consensus.cc:504] T 06d5a3e284c04301b765ec23671f0eee P 88e20b82c7674c67a918e579365cd514 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:39.970758 4904 leader_election.cc:302] T 06d5a3e284c04301b765ec23671f0eee P 88e20b82c7674c67a918e579365cd514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88e20b82c7674c67a918e579365cd514; no voters:
I20260804 14:53:39.971632 4904 leader_election.cc:288] T 06d5a3e284c04301b765ec23671f0eee P 88e20b82c7674c67a918e579365cd514 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260804 14:53:39.971709 4997 raft_consensus.cc:2781] T 06d5a3e284c04301b765ec23671f0eee P 88e20b82c7674c67a918e579365cd514 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:39.971813 4997 raft_consensus.cc:686] T 06d5a3e284c04301b765ec23671f0eee P 88e20b82c7674c67a918e579365cd514 [term 1 LEADER]: Becoming Leader. State: Replica: 88e20b82c7674c67a918e579365cd514, State: Running, Role: LEADER
I20260804 14:53:39.972242 4997 consensus_queue.cc:237] T 06d5a3e284c04301b765ec23671f0eee P 88e20b82c7674c67a918e579365cd514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:39.976922 4702 catalog_manager.cc:5495] T 06d5a3e284c04301b765ec23671f0eee P 88e20b82c7674c67a918e579365cd514 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88e20b82c7674c67a918e579365cd514 (127.24.6.129). New cstate: current_term: 1 leader_uuid: "88e20b82c7674c67a918e579365cd514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } health_report { overall_health: HEALTHY } } }
I20260804 14:53:39.985744 4904 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5a47a19b1204470ba09da1170ea89400"
dest_uuid: "88e20b82c7674c67a918e579365cd514"
from {username='slave'} at 127.0.0.1:51762
I20260804 14:53:39.986066 4904 raft_consensus.cc:482] T 5a47a19b1204470ba09da1170ea89400 P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260804 14:53:39.986136 4904 raft_consensus.cc:3037] T 5a47a19b1204470ba09da1170ea89400 P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:39.987133 4904 raft_consensus.cc:504] T 5a47a19b1204470ba09da1170ea89400 P 88e20b82c7674c67a918e579365cd514 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:39.987365 4904 leader_election.cc:302] T 5a47a19b1204470ba09da1170ea89400 P 88e20b82c7674c67a918e579365cd514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88e20b82c7674c67a918e579365cd514; no voters:
I20260804 14:53:39.987505 4904 leader_election.cc:288] T 5a47a19b1204470ba09da1170ea89400 P 88e20b82c7674c67a918e579365cd514 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260804 14:53:39.987612 4997 raft_consensus.cc:2781] T 5a47a19b1204470ba09da1170ea89400 P 88e20b82c7674c67a918e579365cd514 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:39.987757 4997 raft_consensus.cc:686] T 5a47a19b1204470ba09da1170ea89400 P 88e20b82c7674c67a918e579365cd514 [term 1 LEADER]: Becoming Leader. State: Replica: 88e20b82c7674c67a918e579365cd514, State: Running, Role: LEADER
I20260804 14:53:39.987947 4997 consensus_queue.cc:237] T 5a47a19b1204470ba09da1170ea89400 P 88e20b82c7674c67a918e579365cd514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:39.989773 4702 catalog_manager.cc:5495] T 5a47a19b1204470ba09da1170ea89400 P 88e20b82c7674c67a918e579365cd514 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88e20b82c7674c67a918e579365cd514 (127.24.6.129). New cstate: current_term: 1 leader_uuid: "88e20b82c7674c67a918e579365cd514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } health_report { overall_health: HEALTHY } } }
I20260804 14:53:39.995633 4904 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "17189dbd7cfa4ca9a4b7aecf77e689ff"
dest_uuid: "88e20b82c7674c67a918e579365cd514"
from {username='slave'} at 127.0.0.1:51762
I20260804 14:53:39.995854 4904 raft_consensus.cc:482] T 17189dbd7cfa4ca9a4b7aecf77e689ff P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260804 14:53:39.995944 4904 raft_consensus.cc:3037] T 17189dbd7cfa4ca9a4b7aecf77e689ff P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:39.996908 4904 raft_consensus.cc:504] T 17189dbd7cfa4ca9a4b7aecf77e689ff P 88e20b82c7674c67a918e579365cd514 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:39.997092 4904 leader_election.cc:302] T 17189dbd7cfa4ca9a4b7aecf77e689ff P 88e20b82c7674c67a918e579365cd514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88e20b82c7674c67a918e579365cd514; no voters:
I20260804 14:53:39.997219 4904 leader_election.cc:288] T 17189dbd7cfa4ca9a4b7aecf77e689ff P 88e20b82c7674c67a918e579365cd514 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260804 14:53:39.997327 4997 raft_consensus.cc:2781] T 17189dbd7cfa4ca9a4b7aecf77e689ff P 88e20b82c7674c67a918e579365cd514 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:39.997460 4997 raft_consensus.cc:686] T 17189dbd7cfa4ca9a4b7aecf77e689ff P 88e20b82c7674c67a918e579365cd514 [term 1 LEADER]: Becoming Leader. State: Replica: 88e20b82c7674c67a918e579365cd514, State: Running, Role: LEADER
I20260804 14:53:39.997653 4997 consensus_queue.cc:237] T 17189dbd7cfa4ca9a4b7aecf77e689ff P 88e20b82c7674c67a918e579365cd514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:39.999476 4702 catalog_manager.cc:5495] T 17189dbd7cfa4ca9a4b7aecf77e689ff P 88e20b82c7674c67a918e579365cd514 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88e20b82c7674c67a918e579365cd514 (127.24.6.129). New cstate: current_term: 1 leader_uuid: "88e20b82c7674c67a918e579365cd514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } health_report { overall_health: HEALTHY } } }
I20260804 14:53:40.006644 4904 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2f36d606de4647b695d8b82f70e1f012"
dest_uuid: "88e20b82c7674c67a918e579365cd514"
from {username='slave'} at 127.0.0.1:51762
I20260804 14:53:40.006858 4904 raft_consensus.cc:482] T 2f36d606de4647b695d8b82f70e1f012 P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260804 14:53:40.006920 4904 raft_consensus.cc:3037] T 2f36d606de4647b695d8b82f70e1f012 P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:40.008117 4904 raft_consensus.cc:504] T 2f36d606de4647b695d8b82f70e1f012 P 88e20b82c7674c67a918e579365cd514 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:40.008291 4904 leader_election.cc:302] T 2f36d606de4647b695d8b82f70e1f012 P 88e20b82c7674c67a918e579365cd514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88e20b82c7674c67a918e579365cd514; no voters:
I20260804 14:53:40.008401 4904 leader_election.cc:288] T 2f36d606de4647b695d8b82f70e1f012 P 88e20b82c7674c67a918e579365cd514 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260804 14:53:40.008462 4997 raft_consensus.cc:2781] T 2f36d606de4647b695d8b82f70e1f012 P 88e20b82c7674c67a918e579365cd514 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:40.008560 4997 raft_consensus.cc:686] T 2f36d606de4647b695d8b82f70e1f012 P 88e20b82c7674c67a918e579365cd514 [term 1 LEADER]: Becoming Leader. State: Replica: 88e20b82c7674c67a918e579365cd514, State: Running, Role: LEADER
I20260804 14:53:40.008721 4997 consensus_queue.cc:237] T 2f36d606de4647b695d8b82f70e1f012 P 88e20b82c7674c67a918e579365cd514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
W20260804 14:53:40.009855 4974 tablet.cc:2369] T 2f36d606de4647b695d8b82f70e1f012 P 88e20b82c7674c67a918e579365cd514: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260804 14:53:40.010552 4702 catalog_manager.cc:5495] T 2f36d606de4647b695d8b82f70e1f012 P 88e20b82c7674c67a918e579365cd514 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88e20b82c7674c67a918e579365cd514 (127.24.6.129). New cstate: current_term: 1 leader_uuid: "88e20b82c7674c67a918e579365cd514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } health_report { overall_health: HEALTHY } } }
I20260804 14:53:40.017398 4904 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a941152217e240ffa03ac2c950271763"
dest_uuid: "88e20b82c7674c67a918e579365cd514"
from {username='slave'} at 127.0.0.1:51762
I20260804 14:53:40.017633 4904 raft_consensus.cc:482] T a941152217e240ffa03ac2c950271763 P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260804 14:53:40.017704 4904 raft_consensus.cc:3037] T a941152217e240ffa03ac2c950271763 P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:40.018679 4904 raft_consensus.cc:504] T a941152217e240ffa03ac2c950271763 P 88e20b82c7674c67a918e579365cd514 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:40.018850 4904 leader_election.cc:302] T a941152217e240ffa03ac2c950271763 P 88e20b82c7674c67a918e579365cd514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88e20b82c7674c67a918e579365cd514; no voters:
I20260804 14:53:40.018980 4904 leader_election.cc:288] T a941152217e240ffa03ac2c950271763 P 88e20b82c7674c67a918e579365cd514 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260804 14:53:40.019029 4997 raft_consensus.cc:2781] T a941152217e240ffa03ac2c950271763 P 88e20b82c7674c67a918e579365cd514 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:40.019126 4997 raft_consensus.cc:686] T a941152217e240ffa03ac2c950271763 P 88e20b82c7674c67a918e579365cd514 [term 1 LEADER]: Becoming Leader. State: Replica: 88e20b82c7674c67a918e579365cd514, State: Running, Role: LEADER
I20260804 14:53:40.019299 4997 consensus_queue.cc:237] T a941152217e240ffa03ac2c950271763 P 88e20b82c7674c67a918e579365cd514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:40.021144 4702 catalog_manager.cc:5495] T a941152217e240ffa03ac2c950271763 P 88e20b82c7674c67a918e579365cd514 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88e20b82c7674c67a918e579365cd514 (127.24.6.129). New cstate: current_term: 1 leader_uuid: "88e20b82c7674c67a918e579365cd514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } health_report { overall_health: HEALTHY } } }
I20260804 14:53:40.027662 4904 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cca9b26f4ab5407ebc2ba2c2ea82953e"
dest_uuid: "88e20b82c7674c67a918e579365cd514"
from {username='slave'} at 127.0.0.1:51762
I20260804 14:53:40.027881 4904 raft_consensus.cc:482] T cca9b26f4ab5407ebc2ba2c2ea82953e P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260804 14:53:40.027969 4904 raft_consensus.cc:3037] T cca9b26f4ab5407ebc2ba2c2ea82953e P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:40.029109 4904 raft_consensus.cc:504] T cca9b26f4ab5407ebc2ba2c2ea82953e P 88e20b82c7674c67a918e579365cd514 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:40.029304 4904 leader_election.cc:302] T cca9b26f4ab5407ebc2ba2c2ea82953e P 88e20b82c7674c67a918e579365cd514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88e20b82c7674c67a918e579365cd514; no voters:
I20260804 14:53:40.029505 4904 leader_election.cc:288] T cca9b26f4ab5407ebc2ba2c2ea82953e P 88e20b82c7674c67a918e579365cd514 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260804 14:53:40.029613 4997 raft_consensus.cc:2781] T cca9b26f4ab5407ebc2ba2c2ea82953e P 88e20b82c7674c67a918e579365cd514 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:40.029765 4997 raft_consensus.cc:686] T cca9b26f4ab5407ebc2ba2c2ea82953e P 88e20b82c7674c67a918e579365cd514 [term 1 LEADER]: Becoming Leader. State: Replica: 88e20b82c7674c67a918e579365cd514, State: Running, Role: LEADER
I20260804 14:53:40.029983 4997 consensus_queue.cc:237] T cca9b26f4ab5407ebc2ba2c2ea82953e P 88e20b82c7674c67a918e579365cd514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:40.033403 4702 catalog_manager.cc:5495] T cca9b26f4ab5407ebc2ba2c2ea82953e P 88e20b82c7674c67a918e579365cd514 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88e20b82c7674c67a918e579365cd514 (127.24.6.129). New cstate: current_term: 1 leader_uuid: "88e20b82c7674c67a918e579365cd514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } health_report { overall_health: HEALTHY } } }
I20260804 14:53:40.039862 4904 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a1c71c43dcba46b59519f16d317472fe"
dest_uuid: "88e20b82c7674c67a918e579365cd514"
from {username='slave'} at 127.0.0.1:51762
I20260804 14:53:40.040102 4904 raft_consensus.cc:482] T a1c71c43dcba46b59519f16d317472fe P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260804 14:53:40.040190 4904 raft_consensus.cc:3037] T a1c71c43dcba46b59519f16d317472fe P 88e20b82c7674c67a918e579365cd514 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:40.041178 4904 raft_consensus.cc:504] T a1c71c43dcba46b59519f16d317472fe P 88e20b82c7674c67a918e579365cd514 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:40.041369 4904 leader_election.cc:302] T a1c71c43dcba46b59519f16d317472fe P 88e20b82c7674c67a918e579365cd514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88e20b82c7674c67a918e579365cd514; no voters:
I20260804 14:53:40.041534 4904 leader_election.cc:288] T a1c71c43dcba46b59519f16d317472fe P 88e20b82c7674c67a918e579365cd514 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260804 14:53:40.041671 5012 raft_consensus.cc:2781] T a1c71c43dcba46b59519f16d317472fe P 88e20b82c7674c67a918e579365cd514 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:40.041824 5012 raft_consensus.cc:686] T a1c71c43dcba46b59519f16d317472fe P 88e20b82c7674c67a918e579365cd514 [term 1 LEADER]: Becoming Leader. State: Replica: 88e20b82c7674c67a918e579365cd514, State: Running, Role: LEADER
I20260804 14:53:40.042001 5012 consensus_queue.cc:237] T a1c71c43dcba46b59519f16d317472fe P 88e20b82c7674c67a918e579365cd514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:40.043879 4702 catalog_manager.cc:5495] T a1c71c43dcba46b59519f16d317472fe P 88e20b82c7674c67a918e579365cd514 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88e20b82c7674c67a918e579365cd514 (127.24.6.129). New cstate: current_term: 1 leader_uuid: "88e20b82c7674c67a918e579365cd514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } health_report { overall_health: HEALTHY } } }
I20260804 14:53:40.049885 4872 tablet_service.cc:1514] Processing DeleteTablet for tablet 33d23dc00de74072bccf05a1afa9b62a with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:51762
I20260804 14:53:40.050563 5025 tablet_replica.cc:330] T 33d23dc00de74072bccf05a1afa9b62a P 88e20b82c7674c67a918e579365cd514: stopping tablet replica
I20260804 14:53:40.050961 5025 raft_consensus.cc:2229] T 33d23dc00de74072bccf05a1afa9b62a P 88e20b82c7674c67a918e579365cd514 [term 1 LEADER]: Raft consensus shutting down.
I20260804 14:53:40.051276 5025 raft_consensus.cc:2258] T 33d23dc00de74072bccf05a1afa9b62a P 88e20b82c7674c67a918e579365cd514 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260804 14:53:40.052618 5025 ts_tablet_manager.cc:1905] T 33d23dc00de74072bccf05a1afa9b62a P 88e20b82c7674c67a918e579365cd514: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260804 14:53:40.054971 5025 ts_tablet_manager.cc:1918] T 33d23dc00de74072bccf05a1afa9b62a P 88e20b82c7674c67a918e579365cd514: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260804 14:53:40.055078 5025 log.cc:1196] T 33d23dc00de74072bccf05a1afa9b62a P 88e20b82c7674c67a918e579365cd514: Deleting WAL directory at /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/wals/33d23dc00de74072bccf05a1afa9b62a
I20260804 14:53:40.062112 4872 tablet_service.cc:1514] Processing DeleteTablet for tablet 530fa576e66047d3b2f0838894b7638c with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:51762
I20260804 14:53:40.062399 5025 tablet_replica.cc:330] T 530fa576e66047d3b2f0838894b7638c P 88e20b82c7674c67a918e579365cd514: stopping tablet replica
I20260804 14:53:40.062574 5025 raft_consensus.cc:2229] T 530fa576e66047d3b2f0838894b7638c P 88e20b82c7674c67a918e579365cd514 [term 1 LEADER]: Raft consensus shutting down.
I20260804 14:53:40.062747 5025 raft_consensus.cc:2258] T 530fa576e66047d3b2f0838894b7638c P 88e20b82c7674c67a918e579365cd514 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260804 14:53:40.063557 5025 ts_tablet_manager.cc:1905] T 530fa576e66047d3b2f0838894b7638c P 88e20b82c7674c67a918e579365cd514: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260804 14:53:40.065752 5025 ts_tablet_manager.cc:1918] T 530fa576e66047d3b2f0838894b7638c P 88e20b82c7674c67a918e579365cd514: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260804 14:53:40.065863 5025 log.cc:1196] T 530fa576e66047d3b2f0838894b7638c P 88e20b82c7674c67a918e579365cd514: Deleting WAL directory at /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/wals/530fa576e66047d3b2f0838894b7638c
I20260804 14:53:40.072688 4872 tablet_service.cc:1514] Processing DeleteTablet for tablet 37f31df0068c4c4e921cc713475fd55e with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:51762
I20260804 14:53:40.072980 5025 tablet_replica.cc:330] T 37f31df0068c4c4e921cc713475fd55e P 88e20b82c7674c67a918e579365cd514: stopping tablet replica
I20260804 14:53:40.073169 5025 raft_consensus.cc:2229] T 37f31df0068c4c4e921cc713475fd55e P 88e20b82c7674c67a918e579365cd514 [term 1 LEADER]: Raft consensus shutting down.
I20260804 14:53:40.073355 5025 raft_consensus.cc:2258] T 37f31df0068c4c4e921cc713475fd55e P 88e20b82c7674c67a918e579365cd514 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260804 14:53:40.074143 5025 ts_tablet_manager.cc:1905] T 37f31df0068c4c4e921cc713475fd55e P 88e20b82c7674c67a918e579365cd514: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260804 14:53:40.076294 5025 ts_tablet_manager.cc:1918] T 37f31df0068c4c4e921cc713475fd55e P 88e20b82c7674c67a918e579365cd514: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260804 14:53:40.076400 5025 log.cc:1196] T 37f31df0068c4c4e921cc713475fd55e P 88e20b82c7674c67a918e579365cd514: Deleting WAL directory at /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/wals/37f31df0068c4c4e921cc713475fd55e
I20260804 14:53:40.083256 4872 tablet_service.cc:1514] Processing DeleteTablet for tablet 06d5a3e284c04301b765ec23671f0eee with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:51762
I20260804 14:53:40.083549 5025 tablet_replica.cc:330] T 06d5a3e284c04301b765ec23671f0eee P 88e20b82c7674c67a918e579365cd514: stopping tablet replica
I20260804 14:53:40.083738 5025 raft_consensus.cc:2229] T 06d5a3e284c04301b765ec23671f0eee P 88e20b82c7674c67a918e579365cd514 [term 1 LEADER]: Raft consensus shutting down.
I20260804 14:53:40.083906 5025 raft_consensus.cc:2258] T 06d5a3e284c04301b765ec23671f0eee P 88e20b82c7674c67a918e579365cd514 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260804 14:53:40.084662 5025 ts_tablet_manager.cc:1905] T 06d5a3e284c04301b765ec23671f0eee P 88e20b82c7674c67a918e579365cd514: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260804 14:53:40.087069 5025 ts_tablet_manager.cc:1918] T 06d5a3e284c04301b765ec23671f0eee P 88e20b82c7674c67a918e579365cd514: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260804 14:53:40.087188 5025 log.cc:1196] T 06d5a3e284c04301b765ec23671f0eee P 88e20b82c7674c67a918e579365cd514: Deleting WAL directory at /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/wals/06d5a3e284c04301b765ec23671f0eee
I20260804 14:53:40.094251 4872 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a47a19b1204470ba09da1170ea89400 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:51762
I20260804 14:53:40.094534 5025 tablet_replica.cc:330] T 5a47a19b1204470ba09da1170ea89400 P 88e20b82c7674c67a918e579365cd514: stopping tablet replica
I20260804 14:53:40.094726 5025 raft_consensus.cc:2229] T 5a47a19b1204470ba09da1170ea89400 P 88e20b82c7674c67a918e579365cd514 [term 1 LEADER]: Raft consensus shutting down.
I20260804 14:53:40.094889 5025 raft_consensus.cc:2258] T 5a47a19b1204470ba09da1170ea89400 P 88e20b82c7674c67a918e579365cd514 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260804 14:53:40.095806 5025 ts_tablet_manager.cc:1905] T 5a47a19b1204470ba09da1170ea89400 P 88e20b82c7674c67a918e579365cd514: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260804 14:53:40.098014 5025 ts_tablet_manager.cc:1918] T 5a47a19b1204470ba09da1170ea89400 P 88e20b82c7674c67a918e579365cd514: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260804 14:53:40.098134 5025 log.cc:1196] T 5a47a19b1204470ba09da1170ea89400 P 88e20b82c7674c67a918e579365cd514: Deleting WAL directory at /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/wals/5a47a19b1204470ba09da1170ea89400
I20260804 14:53:40.105295 4904 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "17189dbd7cfa4ca9a4b7aecf77e689ff"
dest_uuid: "88e20b82c7674c67a918e579365cd514"
from {username='slave'} at 127.0.0.1:51762
I20260804 14:53:40.105535 4904 raft_consensus.cc:580] T 17189dbd7cfa4ca9a4b7aecf77e689ff P 88e20b82c7674c67a918e579365cd514 [term 1 LEADER]: Received request to step down
I20260804 14:53:40.105618 4904 raft_consensus.cc:3032] T 17189dbd7cfa4ca9a4b7aecf77e689ff P 88e20b82c7674c67a918e579365cd514 [term 1 LEADER]: Stepping down as leader of term 1
I20260804 14:53:40.105670 4904 raft_consensus.cc:729] T 17189dbd7cfa4ca9a4b7aecf77e689ff P 88e20b82c7674c67a918e579365cd514 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88e20b82c7674c67a918e579365cd514, State: Running, Role: LEADER
I20260804 14:53:40.105842 4904 consensus_queue.cc:260] T 17189dbd7cfa4ca9a4b7aecf77e689ff P 88e20b82c7674c67a918e579365cd514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:40.105942 4904 raft_consensus.cc:3037] T 17189dbd7cfa4ca9a4b7aecf77e689ff P 88e20b82c7674c67a918e579365cd514 [term 1 FOLLOWER]: Advancing to term 2
I20260804 14:53:40.113065 4872 tablet_service.cc:1514] Processing DeleteTablet for tablet 17189dbd7cfa4ca9a4b7aecf77e689ff with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:51762
I20260804 14:53:40.113353 5025 tablet_replica.cc:330] T 17189dbd7cfa4ca9a4b7aecf77e689ff P 88e20b82c7674c67a918e579365cd514: stopping tablet replica
I20260804 14:53:40.113529 5025 raft_consensus.cc:2229] T 17189dbd7cfa4ca9a4b7aecf77e689ff P 88e20b82c7674c67a918e579365cd514 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260804 14:53:40.113708 5025 raft_consensus.cc:2258] T 17189dbd7cfa4ca9a4b7aecf77e689ff P 88e20b82c7674c67a918e579365cd514 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260804 14:53:40.114466 5025 ts_tablet_manager.cc:1905] T 17189dbd7cfa4ca9a4b7aecf77e689ff P 88e20b82c7674c67a918e579365cd514: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260804 14:53:40.116613 5025 ts_tablet_manager.cc:1918] T 17189dbd7cfa4ca9a4b7aecf77e689ff P 88e20b82c7674c67a918e579365cd514: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260804 14:53:40.116717 5025 log.cc:1196] T 17189dbd7cfa4ca9a4b7aecf77e689ff P 88e20b82c7674c67a918e579365cd514: Deleting WAL directory at /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/wals/17189dbd7cfa4ca9a4b7aecf77e689ff
I20260804 14:53:40.123587 4904 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2f36d606de4647b695d8b82f70e1f012"
dest_uuid: "88e20b82c7674c67a918e579365cd514"
from {username='slave'} at 127.0.0.1:51762
I20260804 14:53:40.123811 4904 raft_consensus.cc:580] T 2f36d606de4647b695d8b82f70e1f012 P 88e20b82c7674c67a918e579365cd514 [term 1 LEADER]: Received request to step down
I20260804 14:53:40.123883 4904 raft_consensus.cc:3032] T 2f36d606de4647b695d8b82f70e1f012 P 88e20b82c7674c67a918e579365cd514 [term 1 LEADER]: Stepping down as leader of term 1
I20260804 14:53:40.123924 4904 raft_consensus.cc:729] T 2f36d606de4647b695d8b82f70e1f012 P 88e20b82c7674c67a918e579365cd514 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88e20b82c7674c67a918e579365cd514, State: Running, Role: LEADER
I20260804 14:53:40.124063 4904 consensus_queue.cc:260] T 2f36d606de4647b695d8b82f70e1f012 P 88e20b82c7674c67a918e579365cd514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:40.124193 4904 raft_consensus.cc:3037] T 2f36d606de4647b695d8b82f70e1f012 P 88e20b82c7674c67a918e579365cd514 [term 1 FOLLOWER]: Advancing to term 2
I20260804 14:53:40.131767 4872 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f36d606de4647b695d8b82f70e1f012 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:51762
I20260804 14:53:40.132100 5025 tablet_replica.cc:330] T 2f36d606de4647b695d8b82f70e1f012 P 88e20b82c7674c67a918e579365cd514: stopping tablet replica
I20260804 14:53:40.132292 5025 raft_consensus.cc:2229] T 2f36d606de4647b695d8b82f70e1f012 P 88e20b82c7674c67a918e579365cd514 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260804 14:53:40.132474 5025 raft_consensus.cc:2258] T 2f36d606de4647b695d8b82f70e1f012 P 88e20b82c7674c67a918e579365cd514 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260804 14:53:40.133312 5025 ts_tablet_manager.cc:1905] T 2f36d606de4647b695d8b82f70e1f012 P 88e20b82c7674c67a918e579365cd514: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260804 14:53:40.135637 5025 ts_tablet_manager.cc:1918] T 2f36d606de4647b695d8b82f70e1f012 P 88e20b82c7674c67a918e579365cd514: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260804 14:53:40.135754 5025 log.cc:1196] T 2f36d606de4647b695d8b82f70e1f012 P 88e20b82c7674c67a918e579365cd514: Deleting WAL directory at /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/wals/2f36d606de4647b695d8b82f70e1f012
I20260804 14:53:40.143038 4904 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a941152217e240ffa03ac2c950271763"
dest_uuid: "88e20b82c7674c67a918e579365cd514"
from {username='slave'} at 127.0.0.1:51762
I20260804 14:53:40.143332 4904 raft_consensus.cc:580] T a941152217e240ffa03ac2c950271763 P 88e20b82c7674c67a918e579365cd514 [term 1 LEADER]: Received request to step down
I20260804 14:53:40.143407 4904 raft_consensus.cc:3032] T a941152217e240ffa03ac2c950271763 P 88e20b82c7674c67a918e579365cd514 [term 1 LEADER]: Stepping down as leader of term 1
I20260804 14:53:40.143451 4904 raft_consensus.cc:729] T a941152217e240ffa03ac2c950271763 P 88e20b82c7674c67a918e579365cd514 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88e20b82c7674c67a918e579365cd514, State: Running, Role: LEADER
I20260804 14:53:40.143611 4904 consensus_queue.cc:260] T a941152217e240ffa03ac2c950271763 P 88e20b82c7674c67a918e579365cd514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:40.143715 4904 raft_consensus.cc:3037] T a941152217e240ffa03ac2c950271763 P 88e20b82c7674c67a918e579365cd514 [term 1 FOLLOWER]: Advancing to term 2
I20260804 14:53:40.150866 4872 tablet_service.cc:1514] Processing DeleteTablet for tablet a941152217e240ffa03ac2c950271763 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:51762
I20260804 14:53:40.151289 5025 tablet_replica.cc:330] T a941152217e240ffa03ac2c950271763 P 88e20b82c7674c67a918e579365cd514: stopping tablet replica
I20260804 14:53:40.151458 5025 raft_consensus.cc:2229] T a941152217e240ffa03ac2c950271763 P 88e20b82c7674c67a918e579365cd514 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260804 14:53:40.151628 5025 raft_consensus.cc:2258] T a941152217e240ffa03ac2c950271763 P 88e20b82c7674c67a918e579365cd514 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260804 14:53:40.152465 5025 ts_tablet_manager.cc:1905] T a941152217e240ffa03ac2c950271763 P 88e20b82c7674c67a918e579365cd514: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260804 14:53:40.154485 5025 ts_tablet_manager.cc:1918] T a941152217e240ffa03ac2c950271763 P 88e20b82c7674c67a918e579365cd514: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260804 14:53:40.154605 5025 log.cc:1196] T a941152217e240ffa03ac2c950271763 P 88e20b82c7674c67a918e579365cd514: Deleting WAL directory at /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/wals/a941152217e240ffa03ac2c950271763
I20260804 14:53:40.161530 4904 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cca9b26f4ab5407ebc2ba2c2ea82953e"
dest_uuid: "88e20b82c7674c67a918e579365cd514"
from {username='slave'} at 127.0.0.1:51762
I20260804 14:53:40.161772 4904 raft_consensus.cc:580] T cca9b26f4ab5407ebc2ba2c2ea82953e P 88e20b82c7674c67a918e579365cd514 [term 1 LEADER]: Received request to step down
I20260804 14:53:40.161835 4904 raft_consensus.cc:3032] T cca9b26f4ab5407ebc2ba2c2ea82953e P 88e20b82c7674c67a918e579365cd514 [term 1 LEADER]: Stepping down as leader of term 1
I20260804 14:53:40.161890 4904 raft_consensus.cc:729] T cca9b26f4ab5407ebc2ba2c2ea82953e P 88e20b82c7674c67a918e579365cd514 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88e20b82c7674c67a918e579365cd514, State: Running, Role: LEADER
I20260804 14:53:40.162031 4904 consensus_queue.cc:260] T cca9b26f4ab5407ebc2ba2c2ea82953e P 88e20b82c7674c67a918e579365cd514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:40.162114 4904 raft_consensus.cc:3037] T cca9b26f4ab5407ebc2ba2c2ea82953e P 88e20b82c7674c67a918e579365cd514 [term 1 FOLLOWER]: Advancing to term 2
I20260804 14:53:40.168380 4872 tablet_service.cc:1514] Processing DeleteTablet for tablet cca9b26f4ab5407ebc2ba2c2ea82953e with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:51762
I20260804 14:53:40.168655 5025 tablet_replica.cc:330] T cca9b26f4ab5407ebc2ba2c2ea82953e P 88e20b82c7674c67a918e579365cd514: stopping tablet replica
I20260804 14:53:40.168813 5025 raft_consensus.cc:2229] T cca9b26f4ab5407ebc2ba2c2ea82953e P 88e20b82c7674c67a918e579365cd514 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260804 14:53:40.169023 5025 raft_consensus.cc:2258] T cca9b26f4ab5407ebc2ba2c2ea82953e P 88e20b82c7674c67a918e579365cd514 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260804 14:53:40.169821 5025 ts_tablet_manager.cc:1905] T cca9b26f4ab5407ebc2ba2c2ea82953e P 88e20b82c7674c67a918e579365cd514: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260804 14:53:40.172258 5025 ts_tablet_manager.cc:1918] T cca9b26f4ab5407ebc2ba2c2ea82953e P 88e20b82c7674c67a918e579365cd514: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260804 14:53:40.172375 5025 log.cc:1196] T cca9b26f4ab5407ebc2ba2c2ea82953e P 88e20b82c7674c67a918e579365cd514: Deleting WAL directory at /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/wals/cca9b26f4ab5407ebc2ba2c2ea82953e
I20260804 14:53:40.179692 4904 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a1c71c43dcba46b59519f16d317472fe"
dest_uuid: "88e20b82c7674c67a918e579365cd514"
from {username='slave'} at 127.0.0.1:51762
I20260804 14:53:40.179911 4904 raft_consensus.cc:580] T a1c71c43dcba46b59519f16d317472fe P 88e20b82c7674c67a918e579365cd514 [term 1 LEADER]: Received request to step down
I20260804 14:53:40.179980 4904 raft_consensus.cc:3032] T a1c71c43dcba46b59519f16d317472fe P 88e20b82c7674c67a918e579365cd514 [term 1 LEADER]: Stepping down as leader of term 1
I20260804 14:53:40.180022 4904 raft_consensus.cc:729] T a1c71c43dcba46b59519f16d317472fe P 88e20b82c7674c67a918e579365cd514 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88e20b82c7674c67a918e579365cd514, State: Running, Role: LEADER
I20260804 14:53:40.180161 4904 consensus_queue.cc:260] T a1c71c43dcba46b59519f16d317472fe P 88e20b82c7674c67a918e579365cd514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e20b82c7674c67a918e579365cd514" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 41155 } }
I20260804 14:53:40.180256 4904 raft_consensus.cc:3037] T a1c71c43dcba46b59519f16d317472fe P 88e20b82c7674c67a918e579365cd514 [term 1 FOLLOWER]: Advancing to term 2
I20260804 14:53:40.187497 4872 tablet_service.cc:1514] Processing DeleteTablet for tablet a1c71c43dcba46b59519f16d317472fe with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:51762
I20260804 14:53:40.187791 5025 tablet_replica.cc:330] T a1c71c43dcba46b59519f16d317472fe P 88e20b82c7674c67a918e579365cd514: stopping tablet replica
I20260804 14:53:40.187964 5025 raft_consensus.cc:2229] T a1c71c43dcba46b59519f16d317472fe P 88e20b82c7674c67a918e579365cd514 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260804 14:53:40.188138 5025 raft_consensus.cc:2258] T a1c71c43dcba46b59519f16d317472fe P 88e20b82c7674c67a918e579365cd514 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260804 14:53:40.189144 5025 ts_tablet_manager.cc:1905] T a1c71c43dcba46b59519f16d317472fe P 88e20b82c7674c67a918e579365cd514: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260804 14:53:40.191638 5025 ts_tablet_manager.cc:1918] T a1c71c43dcba46b59519f16d317472fe P 88e20b82c7674c67a918e579365cd514: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260804 14:53:40.191763 5025 log.cc:1196] T a1c71c43dcba46b59519f16d317472fe P 88e20b82c7674c67a918e579365cd514: Deleting WAL directory at /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/wals/a1c71c43dcba46b59519f16d317472fe
I20260804 14:53:40.198704 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 4791
I20260804 14:53:40.211313 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 4668
2026-08-04T14:53:40Z chronyd exiting
[ OK ] NumReplicas/RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric/0 (995 ms)
[ RUN ] NumReplicas/RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric/1
I20260804 14:53:40.244289 24602 test_util.cc:268] Using random seed: -771367109
I20260804 14:53:40.245182 24602 ts_itest-base.cc:112] Starting cluster with:
I20260804 14:53:40.245247 24602 ts_itest-base.cc:113] --------------
I20260804 14:53:40.245268 24602 ts_itest-base.cc:114] 3 tablet servers
I20260804 14:53:40.245296 24602 ts_itest-base.cc:115] 3 replicas per TS
I20260804 14:53:40.245321 24602 ts_itest-base.cc:116] --------------
2026-08-04T14:53:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-04T14:53:40Z Disabled control of system clock
I20260804 14:53:40.257627 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.6.190:37391
--webserver_interface=127.24.6.190
--webserver_port=0
--builtin_ntp_servers=127.24.6.148:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.6.190:37391
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20260804 14:53:40.382091 5036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:40.382486 5036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:40.382561 5036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:40.386842 5036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260804 14:53:40.386962 5036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:40.386998 5036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260804 14:53:40.387034 5036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260804 14:53:40.392325 5036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:43363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.24.6.190:37391
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.6.190:37391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.24.6.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.5036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:40.393793 5036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:40.395429 5036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260804 14:53:40.405257 5036 server_base.cc:1034] running on GCE node
W20260804 14:53:40.405282 5046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:40.405308 5049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:40.405207 5052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:40.406122 5036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:40.407595 5036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:40.408812 5036 hybrid_clock.cc:648] HybridClock initialized: now 1785855220408794 us; error 33 us; skew 500 ppm
I20260804 14:53:40.411145 5036 webserver.cc:460] Webserver started at http://127.24.6.190:40675/ using document root <none> and password file <none>
I20260804 14:53:40.411924 5036 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:40.412002 5036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:40.412269 5036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:40.414620 5036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "64a6fe65fe8b4eabb15a33d0aaaf1c6d"
format_stamp: "Formatted at 2026-08-04 14:53:40 on dist-test-slave-92m1"
I20260804 14:53:40.415374 5036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "64a6fe65fe8b4eabb15a33d0aaaf1c6d"
format_stamp: "Formatted at 2026-08-04 14:53:40 on dist-test-slave-92m1"
I20260804 14:53:40.420135 5036 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260804 14:53:40.423316 5068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:40.424887 5036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260804 14:53:40.425056 5036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
uuid: "64a6fe65fe8b4eabb15a33d0aaaf1c6d"
format_stamp: "Formatted at 2026-08-04 14:53:40 on dist-test-slave-92m1"
I20260804 14:53:40.425213 5036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:40.452190 5036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:40.453112 5036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:40.453361 5036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:40.462075 5036 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.190:37391
I20260804 14:53:40.462085 5140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.190:37391 every 8 connection(s)
I20260804 14:53:40.463271 5036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260804 14:53:40.466194 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 5036
I20260804 14:53:40.466342 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260804 14:53:40.467402 5141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:40.473879 5141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64a6fe65fe8b4eabb15a33d0aaaf1c6d: Bootstrap starting.
I20260804 14:53:40.476410 5141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64a6fe65fe8b4eabb15a33d0aaaf1c6d: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:40.477413 5141 log.cc:824] T 00000000000000000000000000000000 P 64a6fe65fe8b4eabb15a33d0aaaf1c6d: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:40.479727 5141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64a6fe65fe8b4eabb15a33d0aaaf1c6d: No bootstrap required, opened a new log
I20260804 14:53:40.483886 5141 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64a6fe65fe8b4eabb15a33d0aaaf1c6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a6fe65fe8b4eabb15a33d0aaaf1c6d" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 37391 } }
I20260804 14:53:40.484174 5141 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64a6fe65fe8b4eabb15a33d0aaaf1c6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260804 14:53:40.484223 5141 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64a6fe65fe8b4eabb15a33d0aaaf1c6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64a6fe65fe8b4eabb15a33d0aaaf1c6d, State: Initialized, Role: FOLLOWER
I20260804 14:53:40.484688 5141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64a6fe65fe8b4eabb15a33d0aaaf1c6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a6fe65fe8b4eabb15a33d0aaaf1c6d" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 37391 } }
I20260804 14:53:40.484839 5141 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64a6fe65fe8b4eabb15a33d0aaaf1c6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260804 14:53:40.484892 5141 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64a6fe65fe8b4eabb15a33d0aaaf1c6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260804 14:53:40.484972 5141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64a6fe65fe8b4eabb15a33d0aaaf1c6d [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:40.485911 5141 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64a6fe65fe8b4eabb15a33d0aaaf1c6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a6fe65fe8b4eabb15a33d0aaaf1c6d" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 37391 } }
I20260804 14:53:40.486267 5141 leader_election.cc:302] T 00000000000000000000000000000000 P 64a6fe65fe8b4eabb15a33d0aaaf1c6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64a6fe65fe8b4eabb15a33d0aaaf1c6d; no voters:
I20260804 14:53:40.486568 5141 leader_election.cc:288] T 00000000000000000000000000000000 P 64a6fe65fe8b4eabb15a33d0aaaf1c6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260804 14:53:40.486747 5146 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64a6fe65fe8b4eabb15a33d0aaaf1c6d [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:40.487022 5146 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64a6fe65fe8b4eabb15a33d0aaaf1c6d [term 1 LEADER]: Becoming Leader. State: Replica: 64a6fe65fe8b4eabb15a33d0aaaf1c6d, State: Running, Role: LEADER
I20260804 14:53:40.487437 5146 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64a6fe65fe8b4eabb15a33d0aaaf1c6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a6fe65fe8b4eabb15a33d0aaaf1c6d" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 37391 } }
I20260804 14:53:40.488085 5141 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64a6fe65fe8b4eabb15a33d0aaaf1c6d [sys.catalog]: configured and running, proceeding with master startup.
I20260804 14:53:40.489336 5147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64a6fe65fe8b4eabb15a33d0aaaf1c6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64a6fe65fe8b4eabb15a33d0aaaf1c6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a6fe65fe8b4eabb15a33d0aaaf1c6d" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 37391 } } }
I20260804 14:53:40.489498 5147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64a6fe65fe8b4eabb15a33d0aaaf1c6d [sys.catalog]: This master's current role is: LEADER
I20260804 14:53:40.489878 5148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64a6fe65fe8b4eabb15a33d0aaaf1c6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64a6fe65fe8b4eabb15a33d0aaaf1c6d. Latest consensus state: current_term: 1 leader_uuid: "64a6fe65fe8b4eabb15a33d0aaaf1c6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a6fe65fe8b4eabb15a33d0aaaf1c6d" member_type: VOTER last_known_addr { host: "127.24.6.190" port: 37391 } } }
I20260804 14:53:40.489990 5148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64a6fe65fe8b4eabb15a33d0aaaf1c6d [sys.catalog]: This master's current role is: LEADER
W20260804 14:53:40.493853 5165 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 64a6fe65fe8b4eabb15a33d0aaaf1c6d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260804 14:53:40.493983 5165 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260804 14:53:40.494072 5158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260804 14:53:40.494925 5158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260804 14:53:40.501926 5158 catalog_manager.cc:1347] Generated new cluster ID: 36e9186497fb43458cd094a18cfab27f
I20260804 14:53:40.502030 5158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260804 14:53:40.508999 5158 catalog_manager.cc:1370] Generated new certificate authority record
I20260804 14:53:40.510466 5158 catalog_manager.cc:1504] Loading token signing keys...
I20260804 14:53:40.515909 5158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64a6fe65fe8b4eabb15a33d0aaaf1c6d: Generated new TSK 0
I20260804 14:53:40.516873 5158 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260804 14:53:40.524382 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.129:0
--local_ip_for_outbound_sockets=127.24.6.129
--webserver_interface=127.24.6.129
--webserver_port=0
--tserver_master_addrs=127.24.6.190:37391
--builtin_ntp_servers=127.24.6.148:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20260804 14:53:40.657413 5170 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260804 14:53:40.657823 5170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:40.657873 5170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:40.657941 5170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:40.662602 5170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:40.662788 5170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.129
I20260804 14:53:40.668095 5170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:43363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.24.6.129
--webserver_port=0
--tserver_master_addrs=127.24.6.190:37391
--never_fsync=true
--heap_profile_path=/tmp/kudu.5170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.129
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:40.669384 5170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:40.670962 5170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:40.682142 5176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:40.682161 5177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:40.682245 5170 server_base.cc:1034] running on GCE node
W20260804 14:53:40.682505 5179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:40.683005 5170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:40.683846 5170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:40.685070 5170 hybrid_clock.cc:648] HybridClock initialized: now 1785855220685056 us; error 46 us; skew 500 ppm
I20260804 14:53:40.687933 5170 webserver.cc:460] Webserver started at http://127.24.6.129:38955/ using document root <none> and password file <none>
I20260804 14:53:40.688805 5170 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:40.688938 5170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:40.689249 5170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:40.691869 5170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "2fb91ff838c74c0490ce3abaca0e933b"
format_stamp: "Formatted at 2026-08-04 14:53:40 on dist-test-slave-92m1"
I20260804 14:53:40.692646 5170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "2fb91ff838c74c0490ce3abaca0e933b"
format_stamp: "Formatted at 2026-08-04 14:53:40 on dist-test-slave-92m1"
I20260804 14:53:40.697700 5170 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260804 14:53:40.700657 5185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:40.702075 5170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260804 14:53:40.702248 5170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2fb91ff838c74c0490ce3abaca0e933b"
format_stamp: "Formatted at 2026-08-04 14:53:40 on dist-test-slave-92m1"
I20260804 14:53:40.702396 5170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:40.717836 5170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:40.718734 5170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:40.718986 5170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:40.719732 5170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:53:40.720908 5170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:53:40.720988 5170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:40.721065 5170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:53:40.721113 5170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:40.731792 5170 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.129:44579
I20260804 14:53:40.731812 5345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.129:44579 every 8 connection(s)
I20260804 14:53:40.732929 5170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260804 14:53:40.733692 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 5170
I20260804 14:53:40.733872 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260804 14:53:40.736546 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.130:0
--local_ip_for_outbound_sockets=127.24.6.130
--webserver_interface=127.24.6.130
--webserver_port=0
--tserver_master_addrs=127.24.6.190:37391
--builtin_ntp_servers=127.24.6.148:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260804 14:53:40.750279 5346 heartbeater.cc:347] Connected to a master server at 127.24.6.190:37391
I20260804 14:53:40.750586 5346 heartbeater.cc:464] Registering TS with master...
I20260804 14:53:40.751297 5346 heartbeater.cc:511] Master 127.24.6.190:37391 requested a full tablet report, sending...
I20260804 14:53:40.753290 5092 ts_manager.cc:194] Registered new tserver with Master: 2fb91ff838c74c0490ce3abaca0e933b (127.24.6.129:44579)
I20260804 14:53:40.755038 5092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.129:51715
W20260804 14:53:40.868065 5349 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260804 14:53:40.868407 5349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:40.868438 5349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:40.868513 5349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:40.872851 5349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:40.872979 5349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.130
I20260804 14:53:40.878196 5349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:43363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.24.6.130
--webserver_port=0
--tserver_master_addrs=127.24.6.190:37391
--never_fsync=true
--heap_profile_path=/tmp/kudu.5349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.130
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:40.879492 5349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:40.881075 5349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:40.892419 5359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:40.892412 5356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:40.892444 5357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:40.892967 5349 server_base.cc:1034] running on GCE node
I20260804 14:53:40.893505 5349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:40.894359 5349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:40.895638 5349 hybrid_clock.cc:648] HybridClock initialized: now 1785855220895566 us; error 46 us; skew 500 ppm
I20260804 14:53:40.898204 5349 webserver.cc:460] Webserver started at http://127.24.6.130:37447/ using document root <none> and password file <none>
I20260804 14:53:40.898932 5349 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:40.899056 5349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:40.899380 5349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:40.901683 5349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "3e653e9eb883427686d59e1134c442e1"
format_stamp: "Formatted at 2026-08-04 14:53:40 on dist-test-slave-92m1"
I20260804 14:53:40.902422 5349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "3e653e9eb883427686d59e1134c442e1"
format_stamp: "Formatted at 2026-08-04 14:53:40 on dist-test-slave-92m1"
I20260804 14:53:40.907164 5349 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260804 14:53:40.910468 5367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:40.912500 5349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260804 14:53:40.912737 5349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "3e653e9eb883427686d59e1134c442e1"
format_stamp: "Formatted at 2026-08-04 14:53:40 on dist-test-slave-92m1"
I20260804 14:53:40.912894 5349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:40.928362 5349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:40.929175 5349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:40.929364 5349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:40.930115 5349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:53:40.931387 5349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:53:40.931468 5349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:40.931561 5349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:53:40.931599 5349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:40.942339 5349 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.130:38561
I20260804 14:53:40.942360 5520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.130:38561 every 8 connection(s)
I20260804 14:53:40.943578 5349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260804 14:53:40.947629 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 5349
I20260804 14:53:40.947790 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260804 14:53:40.950738 24602 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp1wyor/build/debug/bin/kudu
/tmp/dist-test-taskp1wyor/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.6.131:0
--local_ip_for_outbound_sockets=127.24.6.131
--webserver_interface=127.24.6.131
--webserver_port=0
--tserver_master_addrs=127.24.6.190:37391
--builtin_ntp_servers=127.24.6.148:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260804 14:53:40.957458 5521 heartbeater.cc:347] Connected to a master server at 127.24.6.190:37391
I20260804 14:53:40.957790 5521 heartbeater.cc:464] Registering TS with master...
I20260804 14:53:40.958467 5521 heartbeater.cc:511] Master 127.24.6.190:37391 requested a full tablet report, sending...
I20260804 14:53:40.959859 5092 ts_manager.cc:194] Registered new tserver with Master: 3e653e9eb883427686d59e1134c442e1 (127.24.6.130:38561)
I20260804 14:53:40.960566 5092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.130:60409
W20260804 14:53:41.082124 5524 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260804 14:53:41.082546 5524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260804 14:53:41.082595 5524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260804 14:53:41.082664 5524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260804 14:53:41.086999 5524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260804 14:53:41.087194 5524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.6.131
I20260804 14:53:41.093077 5524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.6.148:43363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.6.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.24.6.131
--webserver_port=0
--tserver_master_addrs=127.24.6.190:37391
--never_fsync=true
--heap_profile_path=/tmp/kudu.5524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.6.131
--log_dir=/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 04 Aug 2026 14:17:11 UTC on ccycloud.quasar-lxlfrn.root.comops.site
build id 4564689
I20260804 14:53:41.094416 5524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260804 14:53:41.096114 5524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260804 14:53:41.107013 5537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:41.106940 5538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260804 14:53:41.106940 5540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260804 14:53:41.107151 5524 server_base.cc:1034] running on GCE node
I20260804 14:53:41.108047 5524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260804 14:53:41.108809 5524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260804 14:53:41.109982 5524 hybrid_clock.cc:648] HybridClock initialized: now 1785855221109957 us; error 46 us; skew 500 ppm
I20260804 14:53:41.112231 5524 webserver.cc:460] Webserver started at http://127.24.6.131:46523/ using document root <none> and password file <none>
I20260804 14:53:41.112962 5524 fs_manager.cc:359] Metadata directory not provided
I20260804 14:53:41.113039 5524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260804 14:53:41.113312 5524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260804 14:53:41.115530 5524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "61d01a203880481b9bed9ab4b6d4cd89"
format_stamp: "Formatted at 2026-08-04 14:53:41 on dist-test-slave-92m1"
I20260804 14:53:41.116139 5524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "61d01a203880481b9bed9ab4b6d4cd89"
format_stamp: "Formatted at 2026-08-04 14:53:41 on dist-test-slave-92m1"
I20260804 14:53:41.120586 5524 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260804 14:53:41.123831 5548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:41.125475 5524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260804 14:53:41.125661 5524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "61d01a203880481b9bed9ab4b6d4cd89"
format_stamp: "Formatted at 2026-08-04 14:53:41 on dist-test-slave-92m1"
I20260804 14:53:41.125803 5524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260804 14:53:41.139050 5524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260804 14:53:41.139974 5524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260804 14:53:41.140195 5524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260804 14:53:41.140849 5524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260804 14:53:41.141919 5524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260804 14:53:41.141993 5524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:41.142067 5524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260804 14:53:41.142109 5524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260804 14:53:41.152370 5524 rpc_server.cc:307] RPC server started. Bound to: 127.24.6.131:44621
I20260804 14:53:41.152388 5709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.6.131:44621 every 8 connection(s)
I20260804 14:53:41.153501 5524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260804 14:53:41.160305 24602 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp1wyor/build/debug/bin/kudu as pid 5524
I20260804 14:53:41.160475 24602 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260804 14:53:41.166097 5710 heartbeater.cc:347] Connected to a master server at 127.24.6.190:37391
I20260804 14:53:41.166419 5710 heartbeater.cc:464] Registering TS with master...
I20260804 14:53:41.167052 5710 heartbeater.cc:511] Master 127.24.6.190:37391 requested a full tablet report, sending...
I20260804 14:53:41.168408 5092 ts_manager.cc:194] Registered new tserver with Master: 61d01a203880481b9bed9ab4b6d4cd89 (127.24.6.131:44621)
I20260804 14:53:41.168983 5092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.6.131:44489
I20260804 14:53:41.175263 24602 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260804 14:53:41.186419 24602 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260804 14:53:41.186501 24602 test_util.cc:268] Using random seed: -770424891
I20260804 14:53:41.200515 5092 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34794:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260804 14:53:41.202255 5092 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260804 14:53:41.227675 5249 tablet_service.cc:1467] Processing CreateTablet for tablet dd390684aa114e3b97056bf52dab4de1 (DEFAULT_TABLE table=test-workload [id=0f7cbb8881594ddca980cf3ed5b04c93]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260804 14:53:41.228613 5599 tablet_service.cc:1467] Processing CreateTablet for tablet c51e30c77aad40f2ad69aa98c09f8c34 (DEFAULT_TABLE table=test-workload [id=0f7cbb8881594ddca980cf3ed5b04c93]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260804 14:53:41.229509 5609 tablet_service.cc:1467] Processing CreateTablet for tablet 335ce7f1cab04f2d8c23b89e30fca53d (DEFAULT_TABLE table=test-workload [id=0f7cbb8881594ddca980cf3ed5b04c93]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260804 14:53:41.229651 5599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c51e30c77aad40f2ad69aa98c09f8c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:41.230114 5248 tablet_service.cc:1467] Processing CreateTablet for tablet 9eda8efcee174a59a84c84e9900b1a85 (DEFAULT_TABLE table=test-workload [id=0f7cbb8881594ddca980cf3ed5b04c93]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260804 14:53:41.229434 5431 tablet_service.cc:1467] Processing CreateTablet for tablet 335ce7f1cab04f2d8c23b89e30fca53d (DEFAULT_TABLE table=test-workload [id=0f7cbb8881594ddca980cf3ed5b04c93]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260804 14:53:41.230651 5246 tablet_service.cc:1467] Processing CreateTablet for tablet be8a881726c9437283de67de4c31559a (DEFAULT_TABLE table=test-workload [id=0f7cbb8881594ddca980cf3ed5b04c93]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260804 14:53:41.230898 5431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 335ce7f1cab04f2d8c23b89e30fca53d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:41.231113 5245 tablet_service.cc:1467] Processing CreateTablet for tablet 7227e11e83004ac7a3079cb8e7a860f5 (DEFAULT_TABLE table=test-workload [id=0f7cbb8881594ddca980cf3ed5b04c93]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260804 14:53:41.231455 5430 tablet_service.cc:1467] Processing CreateTablet for tablet dea55fb16e734405a55313a86ac8e378 (DEFAULT_TABLE table=test-workload [id=0f7cbb8881594ddca980cf3ed5b04c93]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260804 14:53:41.231549 5244 tablet_service.cc:1467] Processing CreateTablet for tablet c51e30c77aad40f2ad69aa98c09f8c34 (DEFAULT_TABLE table=test-workload [id=0f7cbb8881594ddca980cf3ed5b04c93]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260804 14:53:41.231804 5430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dea55fb16e734405a55313a86ac8e378. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:41.232012 5608 tablet_service.cc:1467] Processing CreateTablet for tablet dea55fb16e734405a55313a86ac8e378 (DEFAULT_TABLE table=test-workload [id=0f7cbb8881594ddca980cf3ed5b04c93]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260804 14:53:41.232299 5608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dea55fb16e734405a55313a86ac8e378. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:41.228047 5604 tablet_service.cc:1467] Processing CreateTablet for tablet dd390684aa114e3b97056bf52dab4de1 (DEFAULT_TABLE table=test-workload [id=0f7cbb8881594ddca980cf3ed5b04c93]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260804 14:53:41.233068 5604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd390684aa114e3b97056bf52dab4de1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:41.233637 5429 tablet_service.cc:1467] Processing CreateTablet for tablet 73efc26c877b46828c2b8ccadceaa91a (DEFAULT_TABLE table=test-workload [id=0f7cbb8881594ddca980cf3ed5b04c93]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260804 14:53:41.233956 5429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73efc26c877b46828c2b8ccadceaa91a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:41.235540 5241 tablet_service.cc:1467] Processing CreateTablet for tablet 5dcdea9fba5f4e908350c27a454d97ca (DEFAULT_TABLE table=test-workload [id=0f7cbb8881594ddca980cf3ed5b04c93]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260804 14:53:41.236109 5607 tablet_service.cc:1467] Processing CreateTablet for tablet 73efc26c877b46828c2b8ccadceaa91a (DEFAULT_TABLE table=test-workload [id=0f7cbb8881594ddca980cf3ed5b04c93]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260804 14:53:41.236380 5607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73efc26c877b46828c2b8ccadceaa91a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:41.239573 5739 tablet_bootstrap.cc:492] T dea55fb16e734405a55313a86ac8e378 P 3e653e9eb883427686d59e1134c442e1: Bootstrap starting.
I20260804 14:53:41.239508 5605 tablet_service.cc:1467] Processing CreateTablet for tablet 57ee2bd239614464a99304a04ddc67bf (DEFAULT_TABLE table=test-workload [id=0f7cbb8881594ddca980cf3ed5b04c93]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260804 14:53:41.239848 5605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ee2bd239614464a99304a04ddc67bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:41.240320 5418 tablet_service.cc:1467] Processing CreateTablet for tablet 5dcdea9fba5f4e908350c27a454d97ca (DEFAULT_TABLE table=test-workload [id=0f7cbb8881594ddca980cf3ed5b04c93]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260804 14:53:41.240597 5418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dcdea9fba5f4e908350c27a454d97ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:41.242933 5739 tablet_bootstrap.cc:654] T dea55fb16e734405a55313a86ac8e378 P 3e653e9eb883427686d59e1134c442e1: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:41.243094 5420 tablet_service.cc:1467] Processing CreateTablet for tablet c51e30c77aad40f2ad69aa98c09f8c34 (DEFAULT_TABLE table=test-workload [id=0f7cbb8881594ddca980cf3ed5b04c93]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260804 14:53:41.243456 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c51e30c77aad40f2ad69aa98c09f8c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:41.243662 5603 tablet_service.cc:1467] Processing CreateTablet for tablet 9eda8efcee174a59a84c84e9900b1a85 (DEFAULT_TABLE table=test-workload [id=0f7cbb8881594ddca980cf3ed5b04c93]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260804 14:53:41.244035 5603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eda8efcee174a59a84c84e9900b1a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:41.244017 5739 log.cc:824] T dea55fb16e734405a55313a86ac8e378 P 3e653e9eb883427686d59e1134c442e1: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:41.246155 5739 tablet_bootstrap.cc:492] T dea55fb16e734405a55313a86ac8e378 P 3e653e9eb883427686d59e1134c442e1: No bootstrap required, opened a new log
I20260804 14:53:41.246333 5739 ts_tablet_manager.cc:1397] T dea55fb16e734405a55313a86ac8e378 P 3e653e9eb883427686d59e1134c442e1: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260804 14:53:41.246460 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 7227e11e83004ac7a3079cb8e7a860f5 (DEFAULT_TABLE table=test-workload [id=0f7cbb8881594ddca980cf3ed5b04c93]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260804 14:53:41.246824 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7227e11e83004ac7a3079cb8e7a860f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:41.247659 5602 tablet_service.cc:1467] Processing CreateTablet for tablet be8a881726c9437283de67de4c31559a (DEFAULT_TABLE table=test-workload [id=0f7cbb8881594ddca980cf3ed5b04c93]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260804 14:53:41.247711 5740 tablet_bootstrap.cc:492] T 57ee2bd239614464a99304a04ddc67bf P 61d01a203880481b9bed9ab4b6d4cd89: Bootstrap starting.
I20260804 14:53:41.248003 5602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be8a881726c9437283de67de4c31559a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:41.249893 5422 tablet_service.cc:1467] Processing CreateTablet for tablet be8a881726c9437283de67de4c31559a (DEFAULT_TABLE table=test-workload [id=0f7cbb8881594ddca980cf3ed5b04c93]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260804 14:53:41.250214 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be8a881726c9437283de67de4c31559a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:41.250329 5740 tablet_bootstrap.cc:654] T 57ee2bd239614464a99304a04ddc67bf P 61d01a203880481b9bed9ab4b6d4cd89: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:41.250429 5739 raft_consensus.cc:348] T dea55fb16e734405a55313a86ac8e378 P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.250751 5739 raft_consensus.cc:729] T dea55fb16e734405a55313a86ac8e378 P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e653e9eb883427686d59e1134c442e1, State: Initialized, Role: FOLLOWER
I20260804 14:53:41.251107 5739 consensus_queue.cc:260] T dea55fb16e734405a55313a86ac8e378 P 3e653e9eb883427686d59e1134c442e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.251283 5740 log.cc:824] T 57ee2bd239614464a99304a04ddc67bf P 61d01a203880481b9bed9ab4b6d4cd89: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:41.251636 5600 tablet_service.cc:1467] Processing CreateTablet for tablet 7227e11e83004ac7a3079cb8e7a860f5 (DEFAULT_TABLE table=test-workload [id=0f7cbb8881594ddca980cf3ed5b04c93]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260804 14:53:41.251973 5600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7227e11e83004ac7a3079cb8e7a860f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:41.252350 5739 ts_tablet_manager.cc:1428] T dea55fb16e734405a55313a86ac8e378 P 3e653e9eb883427686d59e1134c442e1: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260804 14:53:41.252755 5521 heartbeater.cc:503] Master 127.24.6.190:37391 was elected leader, sending a full tablet report...
I20260804 14:53:41.253504 5740 tablet_bootstrap.cc:492] T 57ee2bd239614464a99304a04ddc67bf P 61d01a203880481b9bed9ab4b6d4cd89: No bootstrap required, opened a new log
I20260804 14:53:41.253723 5740 ts_tablet_manager.cc:1397] T 57ee2bd239614464a99304a04ddc67bf P 61d01a203880481b9bed9ab4b6d4cd89: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260804 14:53:41.253712 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 9eda8efcee174a59a84c84e9900b1a85 (DEFAULT_TABLE table=test-workload [id=0f7cbb8881594ddca980cf3ed5b04c93]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260804 14:53:41.254040 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eda8efcee174a59a84c84e9900b1a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:41.254243 5739 tablet_bootstrap.cc:492] T 335ce7f1cab04f2d8c23b89e30fca53d P 3e653e9eb883427686d59e1134c442e1: Bootstrap starting.
I20260804 14:53:41.255331 5739 tablet_bootstrap.cc:654] T 335ce7f1cab04f2d8c23b89e30fca53d P 3e653e9eb883427686d59e1134c442e1: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:41.255676 5598 tablet_service.cc:1467] Processing CreateTablet for tablet 5dcdea9fba5f4e908350c27a454d97ca (DEFAULT_TABLE table=test-workload [id=0f7cbb8881594ddca980cf3ed5b04c93]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260804 14:53:41.255977 5598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dcdea9fba5f4e908350c27a454d97ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:41.256549 5739 tablet_bootstrap.cc:492] T 335ce7f1cab04f2d8c23b89e30fca53d P 3e653e9eb883427686d59e1134c442e1: No bootstrap required, opened a new log
I20260804 14:53:41.256631 5739 ts_tablet_manager.cc:1397] T 335ce7f1cab04f2d8c23b89e30fca53d P 3e653e9eb883427686d59e1134c442e1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260804 14:53:41.256927 5426 tablet_service.cc:1467] Processing CreateTablet for tablet dd390684aa114e3b97056bf52dab4de1 (DEFAULT_TABLE table=test-workload [id=0f7cbb8881594ddca980cf3ed5b04c93]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260804 14:53:41.257230 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd390684aa114e3b97056bf52dab4de1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:41.257311 5739 raft_consensus.cc:348] T 335ce7f1cab04f2d8c23b89e30fca53d P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.257423 5739 raft_consensus.cc:729] T 335ce7f1cab04f2d8c23b89e30fca53d P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e653e9eb883427686d59e1134c442e1, State: Initialized, Role: FOLLOWER
I20260804 14:53:41.257519 5739 consensus_queue.cc:260] T 335ce7f1cab04f2d8c23b89e30fca53d P 3e653e9eb883427686d59e1134c442e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.257889 5739 ts_tablet_manager.cc:1428] T 335ce7f1cab04f2d8c23b89e30fca53d P 3e653e9eb883427686d59e1134c442e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260804 14:53:41.258059 5739 tablet_bootstrap.cc:492] T 73efc26c877b46828c2b8ccadceaa91a P 3e653e9eb883427686d59e1134c442e1: Bootstrap starting.
I20260804 14:53:41.258229 5740 raft_consensus.cc:348] T 57ee2bd239614464a99304a04ddc67bf P 61d01a203880481b9bed9ab4b6d4cd89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.258534 5740 raft_consensus.cc:729] T 57ee2bd239614464a99304a04ddc67bf P 61d01a203880481b9bed9ab4b6d4cd89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61d01a203880481b9bed9ab4b6d4cd89, State: Initialized, Role: FOLLOWER
I20260804 14:53:41.258893 5740 consensus_queue.cc:260] T 57ee2bd239614464a99304a04ddc67bf P 61d01a203880481b9bed9ab4b6d4cd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.259033 5739 tablet_bootstrap.cc:654] T 73efc26c877b46828c2b8ccadceaa91a P 3e653e9eb883427686d59e1134c442e1: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:41.259531 5428 tablet_service.cc:1467] Processing CreateTablet for tablet 57ee2bd239614464a99304a04ddc67bf (DEFAULT_TABLE table=test-workload [id=0f7cbb8881594ddca980cf3ed5b04c93]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260804 14:53:41.259847 5428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ee2bd239614464a99304a04ddc67bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:41.260210 5740 ts_tablet_manager.cc:1428] T 57ee2bd239614464a99304a04ddc67bf P 61d01a203880481b9bed9ab4b6d4cd89: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
I20260804 14:53:41.260353 5739 tablet_bootstrap.cc:492] T 73efc26c877b46828c2b8ccadceaa91a P 3e653e9eb883427686d59e1134c442e1: No bootstrap required, opened a new log
I20260804 14:53:41.260422 5739 ts_tablet_manager.cc:1397] T 73efc26c877b46828c2b8ccadceaa91a P 3e653e9eb883427686d59e1134c442e1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260804 14:53:41.260630 5740 tablet_bootstrap.cc:492] T 9eda8efcee174a59a84c84e9900b1a85 P 61d01a203880481b9bed9ab4b6d4cd89: Bootstrap starting.
I20260804 14:53:41.260676 5710 heartbeater.cc:503] Master 127.24.6.190:37391 was elected leader, sending a full tablet report...
I20260804 14:53:41.260851 5739 raft_consensus.cc:348] T 73efc26c877b46828c2b8ccadceaa91a P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.260953 5739 raft_consensus.cc:729] T 73efc26c877b46828c2b8ccadceaa91a P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e653e9eb883427686d59e1134c442e1, State: Initialized, Role: FOLLOWER
I20260804 14:53:41.261042 5739 consensus_queue.cc:260] T 73efc26c877b46828c2b8ccadceaa91a P 3e653e9eb883427686d59e1134c442e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.261618 5739 ts_tablet_manager.cc:1428] T 73efc26c877b46828c2b8ccadceaa91a P 3e653e9eb883427686d59e1134c442e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260804 14:53:41.261818 5739 tablet_bootstrap.cc:492] T 5dcdea9fba5f4e908350c27a454d97ca P 3e653e9eb883427686d59e1134c442e1: Bootstrap starting.
I20260804 14:53:41.261850 5740 tablet_bootstrap.cc:654] T 9eda8efcee174a59a84c84e9900b1a85 P 61d01a203880481b9bed9ab4b6d4cd89: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:41.262538 5256 tablet_service.cc:1467] Processing CreateTablet for tablet 335ce7f1cab04f2d8c23b89e30fca53d (DEFAULT_TABLE table=test-workload [id=0f7cbb8881594ddca980cf3ed5b04c93]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260804 14:53:41.262805 5739 tablet_bootstrap.cc:654] T 5dcdea9fba5f4e908350c27a454d97ca P 3e653e9eb883427686d59e1134c442e1: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:41.262972 5253 tablet_service.cc:1467] Processing CreateTablet for tablet dea55fb16e734405a55313a86ac8e378 (DEFAULT_TABLE table=test-workload [id=0f7cbb8881594ddca980cf3ed5b04c93]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260804 14:53:41.263370 5609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 335ce7f1cab04f2d8c23b89e30fca53d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:41.263711 5740 tablet_bootstrap.cc:492] T 9eda8efcee174a59a84c84e9900b1a85 P 61d01a203880481b9bed9ab4b6d4cd89: No bootstrap required, opened a new log
I20260804 14:53:41.263792 5740 ts_tablet_manager.cc:1397] T 9eda8efcee174a59a84c84e9900b1a85 P 61d01a203880481b9bed9ab4b6d4cd89: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260804 14:53:41.263944 5739 tablet_bootstrap.cc:492] T 5dcdea9fba5f4e908350c27a454d97ca P 3e653e9eb883427686d59e1134c442e1: No bootstrap required, opened a new log
I20260804 14:53:41.264010 5739 ts_tablet_manager.cc:1397] T 5dcdea9fba5f4e908350c27a454d97ca P 3e653e9eb883427686d59e1134c442e1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260804 14:53:41.264429 5740 raft_consensus.cc:348] T 9eda8efcee174a59a84c84e9900b1a85 P 61d01a203880481b9bed9ab4b6d4cd89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.264397 5739 raft_consensus.cc:348] T 5dcdea9fba5f4e908350c27a454d97ca P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.264523 5740 raft_consensus.cc:729] T 9eda8efcee174a59a84c84e9900b1a85 P 61d01a203880481b9bed9ab4b6d4cd89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61d01a203880481b9bed9ab4b6d4cd89, State: Initialized, Role: FOLLOWER
I20260804 14:53:41.264523 5739 raft_consensus.cc:729] T 5dcdea9fba5f4e908350c27a454d97ca P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e653e9eb883427686d59e1134c442e1, State: Initialized, Role: FOLLOWER
I20260804 14:53:41.264616 5740 consensus_queue.cc:260] T 9eda8efcee174a59a84c84e9900b1a85 P 61d01a203880481b9bed9ab4b6d4cd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.264616 5739 consensus_queue.cc:260] T 5dcdea9fba5f4e908350c27a454d97ca P 3e653e9eb883427686d59e1134c442e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.264870 5740 ts_tablet_manager.cc:1428] T 9eda8efcee174a59a84c84e9900b1a85 P 61d01a203880481b9bed9ab4b6d4cd89: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260804 14:53:41.264870 5739 ts_tablet_manager.cc:1428] T 5dcdea9fba5f4e908350c27a454d97ca P 3e653e9eb883427686d59e1134c442e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260804 14:53:41.265027 5739 tablet_bootstrap.cc:492] T c51e30c77aad40f2ad69aa98c09f8c34 P 3e653e9eb883427686d59e1134c442e1: Bootstrap starting.
I20260804 14:53:41.265033 5740 tablet_bootstrap.cc:492] T be8a881726c9437283de67de4c31559a P 61d01a203880481b9bed9ab4b6d4cd89: Bootstrap starting.
I20260804 14:53:41.266010 5740 tablet_bootstrap.cc:654] T be8a881726c9437283de67de4c31559a P 61d01a203880481b9bed9ab4b6d4cd89: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:41.266003 5739 tablet_bootstrap.cc:654] T c51e30c77aad40f2ad69aa98c09f8c34 P 3e653e9eb883427686d59e1134c442e1: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:41.266929 5252 tablet_service.cc:1467] Processing CreateTablet for tablet 73efc26c877b46828c2b8ccadceaa91a (DEFAULT_TABLE table=test-workload [id=0f7cbb8881594ddca980cf3ed5b04c93]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260804 14:53:41.267292 5740 tablet_bootstrap.cc:492] T be8a881726c9437283de67de4c31559a P 61d01a203880481b9bed9ab4b6d4cd89: No bootstrap required, opened a new log
I20260804 14:53:41.267292 5739 tablet_bootstrap.cc:492] T c51e30c77aad40f2ad69aa98c09f8c34 P 3e653e9eb883427686d59e1134c442e1: No bootstrap required, opened a new log
I20260804 14:53:41.267367 5740 ts_tablet_manager.cc:1397] T be8a881726c9437283de67de4c31559a P 61d01a203880481b9bed9ab4b6d4cd89: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260804 14:53:41.267367 5739 ts_tablet_manager.cc:1397] T c51e30c77aad40f2ad69aa98c09f8c34 P 3e653e9eb883427686d59e1134c442e1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260804 14:53:41.267419 5250 tablet_service.cc:1467] Processing CreateTablet for tablet 57ee2bd239614464a99304a04ddc67bf (DEFAULT_TABLE table=test-workload [id=0f7cbb8881594ddca980cf3ed5b04c93]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260804 14:53:41.267809 5739 raft_consensus.cc:348] T c51e30c77aad40f2ad69aa98c09f8c34 P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.267809 5740 raft_consensus.cc:348] T be8a881726c9437283de67de4c31559a P 61d01a203880481b9bed9ab4b6d4cd89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.267925 5739 raft_consensus.cc:729] T c51e30c77aad40f2ad69aa98c09f8c34 P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e653e9eb883427686d59e1134c442e1, State: Initialized, Role: FOLLOWER
I20260804 14:53:41.267925 5740 raft_consensus.cc:729] T be8a881726c9437283de67de4c31559a P 61d01a203880481b9bed9ab4b6d4cd89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61d01a203880481b9bed9ab4b6d4cd89, State: Initialized, Role: FOLLOWER
I20260804 14:53:41.268023 5739 consensus_queue.cc:260] T c51e30c77aad40f2ad69aa98c09f8c34 P 3e653e9eb883427686d59e1134c442e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.268028 5740 consensus_queue.cc:260] T be8a881726c9437283de67de4c31559a P 61d01a203880481b9bed9ab4b6d4cd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.268285 5739 ts_tablet_manager.cc:1428] T c51e30c77aad40f2ad69aa98c09f8c34 P 3e653e9eb883427686d59e1134c442e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260804 14:53:41.268352 5740 ts_tablet_manager.cc:1428] T be8a881726c9437283de67de4c31559a P 61d01a203880481b9bed9ab4b6d4cd89: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260804 14:53:41.268448 5739 tablet_bootstrap.cc:492] T 7227e11e83004ac7a3079cb8e7a860f5 P 3e653e9eb883427686d59e1134c442e1: Bootstrap starting.
I20260804 14:53:41.268497 5740 tablet_bootstrap.cc:492] T 7227e11e83004ac7a3079cb8e7a860f5 P 61d01a203880481b9bed9ab4b6d4cd89: Bootstrap starting.
I20260804 14:53:41.269532 5739 tablet_bootstrap.cc:654] T 7227e11e83004ac7a3079cb8e7a860f5 P 3e653e9eb883427686d59e1134c442e1: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:41.269874 5740 tablet_bootstrap.cc:654] T 7227e11e83004ac7a3079cb8e7a860f5 P 61d01a203880481b9bed9ab4b6d4cd89: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:41.270869 5739 tablet_bootstrap.cc:492] T 7227e11e83004ac7a3079cb8e7a860f5 P 3e653e9eb883427686d59e1134c442e1: No bootstrap required, opened a new log
I20260804 14:53:41.270956 5739 ts_tablet_manager.cc:1397] T 7227e11e83004ac7a3079cb8e7a860f5 P 3e653e9eb883427686d59e1134c442e1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260804 14:53:41.271080 5740 tablet_bootstrap.cc:492] T 7227e11e83004ac7a3079cb8e7a860f5 P 61d01a203880481b9bed9ab4b6d4cd89: No bootstrap required, opened a new log
I20260804 14:53:41.271164 5740 ts_tablet_manager.cc:1397] T 7227e11e83004ac7a3079cb8e7a860f5 P 61d01a203880481b9bed9ab4b6d4cd89: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260804 14:53:41.271620 5739 raft_consensus.cc:348] T 7227e11e83004ac7a3079cb8e7a860f5 P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.271620 5740 raft_consensus.cc:348] T 7227e11e83004ac7a3079cb8e7a860f5 P 61d01a203880481b9bed9ab4b6d4cd89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.271744 5739 raft_consensus.cc:729] T 7227e11e83004ac7a3079cb8e7a860f5 P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e653e9eb883427686d59e1134c442e1, State: Initialized, Role: FOLLOWER
I20260804 14:53:41.271744 5740 raft_consensus.cc:729] T 7227e11e83004ac7a3079cb8e7a860f5 P 61d01a203880481b9bed9ab4b6d4cd89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61d01a203880481b9bed9ab4b6d4cd89, State: Initialized, Role: FOLLOWER
I20260804 14:53:41.271838 5739 consensus_queue.cc:260] T 7227e11e83004ac7a3079cb8e7a860f5 P 3e653e9eb883427686d59e1134c442e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.271838 5740 consensus_queue.cc:260] T 7227e11e83004ac7a3079cb8e7a860f5 P 61d01a203880481b9bed9ab4b6d4cd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.272102 5739 ts_tablet_manager.cc:1428] T 7227e11e83004ac7a3079cb8e7a860f5 P 3e653e9eb883427686d59e1134c442e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260804 14:53:41.272109 5740 ts_tablet_manager.cc:1428] T 7227e11e83004ac7a3079cb8e7a860f5 P 61d01a203880481b9bed9ab4b6d4cd89: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260804 14:53:41.272264 5740 tablet_bootstrap.cc:492] T 5dcdea9fba5f4e908350c27a454d97ca P 61d01a203880481b9bed9ab4b6d4cd89: Bootstrap starting.
I20260804 14:53:41.272264 5739 tablet_bootstrap.cc:492] T be8a881726c9437283de67de4c31559a P 3e653e9eb883427686d59e1134c442e1: Bootstrap starting.
I20260804 14:53:41.273248 5740 tablet_bootstrap.cc:654] T 5dcdea9fba5f4e908350c27a454d97ca P 61d01a203880481b9bed9ab4b6d4cd89: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:41.273248 5739 tablet_bootstrap.cc:654] T be8a881726c9437283de67de4c31559a P 3e653e9eb883427686d59e1134c442e1: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:41.274773 5740 tablet_bootstrap.cc:492] T 5dcdea9fba5f4e908350c27a454d97ca P 61d01a203880481b9bed9ab4b6d4cd89: No bootstrap required, opened a new log
I20260804 14:53:41.274857 5740 ts_tablet_manager.cc:1397] T 5dcdea9fba5f4e908350c27a454d97ca P 61d01a203880481b9bed9ab4b6d4cd89: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260804 14:53:41.275415 5740 raft_consensus.cc:348] T 5dcdea9fba5f4e908350c27a454d97ca P 61d01a203880481b9bed9ab4b6d4cd89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.275552 5740 raft_consensus.cc:729] T 5dcdea9fba5f4e908350c27a454d97ca P 61d01a203880481b9bed9ab4b6d4cd89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61d01a203880481b9bed9ab4b6d4cd89, State: Initialized, Role: FOLLOWER
I20260804 14:53:41.275656 5740 consensus_queue.cc:260] T 5dcdea9fba5f4e908350c27a454d97ca P 61d01a203880481b9bed9ab4b6d4cd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.275862 5739 tablet_bootstrap.cc:492] T be8a881726c9437283de67de4c31559a P 3e653e9eb883427686d59e1134c442e1: No bootstrap required, opened a new log
I20260804 14:53:41.275952 5739 ts_tablet_manager.cc:1397] T be8a881726c9437283de67de4c31559a P 3e653e9eb883427686d59e1134c442e1: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260804 14:53:41.275987 5740 ts_tablet_manager.cc:1428] T 5dcdea9fba5f4e908350c27a454d97ca P 61d01a203880481b9bed9ab4b6d4cd89: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260804 14:53:41.276155 5740 tablet_bootstrap.cc:492] T 335ce7f1cab04f2d8c23b89e30fca53d P 61d01a203880481b9bed9ab4b6d4cd89: Bootstrap starting.
I20260804 14:53:41.276383 5739 raft_consensus.cc:348] T be8a881726c9437283de67de4c31559a P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.276497 5739 raft_consensus.cc:729] T be8a881726c9437283de67de4c31559a P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e653e9eb883427686d59e1134c442e1, State: Initialized, Role: FOLLOWER
I20260804 14:53:41.276594 5739 consensus_queue.cc:260] T be8a881726c9437283de67de4c31559a P 3e653e9eb883427686d59e1134c442e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.276861 5739 ts_tablet_manager.cc:1428] T be8a881726c9437283de67de4c31559a P 3e653e9eb883427686d59e1134c442e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260804 14:53:41.277241 5740 tablet_bootstrap.cc:654] T 335ce7f1cab04f2d8c23b89e30fca53d P 61d01a203880481b9bed9ab4b6d4cd89: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:41.277369 5739 tablet_bootstrap.cc:492] T 9eda8efcee174a59a84c84e9900b1a85 P 3e653e9eb883427686d59e1134c442e1: Bootstrap starting.
I20260804 14:53:41.278437 5249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd390684aa114e3b97056bf52dab4de1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:41.278564 5739 tablet_bootstrap.cc:654] T 9eda8efcee174a59a84c84e9900b1a85 P 3e653e9eb883427686d59e1134c442e1: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:41.280310 5739 tablet_bootstrap.cc:492] T 9eda8efcee174a59a84c84e9900b1a85 P 3e653e9eb883427686d59e1134c442e1: No bootstrap required, opened a new log
I20260804 14:53:41.280419 5739 ts_tablet_manager.cc:1397] T 9eda8efcee174a59a84c84e9900b1a85 P 3e653e9eb883427686d59e1134c442e1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20260804 14:53:41.280870 5253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dea55fb16e734405a55313a86ac8e378. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:41.281028 5740 tablet_bootstrap.cc:492] T 335ce7f1cab04f2d8c23b89e30fca53d P 61d01a203880481b9bed9ab4b6d4cd89: No bootstrap required, opened a new log
I20260804 14:53:41.280951 5739 raft_consensus.cc:348] T 9eda8efcee174a59a84c84e9900b1a85 P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.281080 5739 raft_consensus.cc:729] T 9eda8efcee174a59a84c84e9900b1a85 P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e653e9eb883427686d59e1134c442e1, State: Initialized, Role: FOLLOWER
I20260804 14:53:41.281109 5740 ts_tablet_manager.cc:1397] T 335ce7f1cab04f2d8c23b89e30fca53d P 61d01a203880481b9bed9ab4b6d4cd89: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260804 14:53:41.281179 5739 consensus_queue.cc:260] T 9eda8efcee174a59a84c84e9900b1a85 P 3e653e9eb883427686d59e1134c442e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.281446 5739 ts_tablet_manager.cc:1428] T 9eda8efcee174a59a84c84e9900b1a85 P 3e653e9eb883427686d59e1134c442e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260804 14:53:41.281622 5739 tablet_bootstrap.cc:492] T 57ee2bd239614464a99304a04ddc67bf P 3e653e9eb883427686d59e1134c442e1: Bootstrap starting.
I20260804 14:53:41.281562 5740 raft_consensus.cc:348] T 335ce7f1cab04f2d8c23b89e30fca53d P 61d01a203880481b9bed9ab4b6d4cd89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.281673 5740 raft_consensus.cc:729] T 335ce7f1cab04f2d8c23b89e30fca53d P 61d01a203880481b9bed9ab4b6d4cd89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61d01a203880481b9bed9ab4b6d4cd89, State: Initialized, Role: FOLLOWER
I20260804 14:53:41.281764 5740 consensus_queue.cc:260] T 335ce7f1cab04f2d8c23b89e30fca53d P 61d01a203880481b9bed9ab4b6d4cd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.281956 5244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c51e30c77aad40f2ad69aa98c09f8c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:41.282611 5739 tablet_bootstrap.cc:654] T 57ee2bd239614464a99304a04ddc67bf P 3e653e9eb883427686d59e1134c442e1: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:41.282768 5246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be8a881726c9437283de67de4c31559a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:41.283187 5250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ee2bd239614464a99304a04ddc67bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:41.284011 5740 ts_tablet_manager.cc:1428] T 335ce7f1cab04f2d8c23b89e30fca53d P 61d01a203880481b9bed9ab4b6d4cd89: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260804 14:53:41.284297 5740 tablet_bootstrap.cc:492] T c51e30c77aad40f2ad69aa98c09f8c34 P 61d01a203880481b9bed9ab4b6d4cd89: Bootstrap starting.
I20260804 14:53:41.284695 5739 tablet_bootstrap.cc:492] T 57ee2bd239614464a99304a04ddc67bf P 3e653e9eb883427686d59e1134c442e1: No bootstrap required, opened a new log
I20260804 14:53:41.284794 5739 ts_tablet_manager.cc:1397] T 57ee2bd239614464a99304a04ddc67bf P 3e653e9eb883427686d59e1134c442e1: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260804 14:53:41.285359 5739 raft_consensus.cc:348] T 57ee2bd239614464a99304a04ddc67bf P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.285509 5739 raft_consensus.cc:729] T 57ee2bd239614464a99304a04ddc67bf P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e653e9eb883427686d59e1134c442e1, State: Initialized, Role: FOLLOWER
I20260804 14:53:41.285616 5739 consensus_queue.cc:260] T 57ee2bd239614464a99304a04ddc67bf P 3e653e9eb883427686d59e1134c442e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.285812 5740 tablet_bootstrap.cc:654] T c51e30c77aad40f2ad69aa98c09f8c34 P 61d01a203880481b9bed9ab4b6d4cd89: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:41.286085 5739 ts_tablet_manager.cc:1428] T 57ee2bd239614464a99304a04ddc67bf P 3e653e9eb883427686d59e1134c442e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260804 14:53:41.286293 5739 tablet_bootstrap.cc:492] T dd390684aa114e3b97056bf52dab4de1 P 3e653e9eb883427686d59e1134c442e1: Bootstrap starting.
I20260804 14:53:41.286728 5245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7227e11e83004ac7a3079cb8e7a860f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:41.287351 5739 tablet_bootstrap.cc:654] T dd390684aa114e3b97056bf52dab4de1 P 3e653e9eb883427686d59e1134c442e1: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:41.288499 5256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 335ce7f1cab04f2d8c23b89e30fca53d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:41.289749 5739 tablet_bootstrap.cc:492] T dd390684aa114e3b97056bf52dab4de1 P 3e653e9eb883427686d59e1134c442e1: No bootstrap required, opened a new log
I20260804 14:53:41.289844 5739 ts_tablet_manager.cc:1397] T dd390684aa114e3b97056bf52dab4de1 P 3e653e9eb883427686d59e1134c442e1: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260804 14:53:41.290218 5248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eda8efcee174a59a84c84e9900b1a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:41.290390 5739 raft_consensus.cc:348] T dd390684aa114e3b97056bf52dab4de1 P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.290534 5739 raft_consensus.cc:729] T dd390684aa114e3b97056bf52dab4de1 P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e653e9eb883427686d59e1134c442e1, State: Initialized, Role: FOLLOWER
I20260804 14:53:41.290653 5739 consensus_queue.cc:260] T dd390684aa114e3b97056bf52dab4de1 P 3e653e9eb883427686d59e1134c442e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.290949 5739 ts_tablet_manager.cc:1428] T dd390684aa114e3b97056bf52dab4de1 P 3e653e9eb883427686d59e1134c442e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260804 14:53:41.291432 5241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dcdea9fba5f4e908350c27a454d97ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:41.292794 5252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73efc26c877b46828c2b8ccadceaa91a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260804 14:53:41.296031 5740 tablet_bootstrap.cc:492] T c51e30c77aad40f2ad69aa98c09f8c34 P 61d01a203880481b9bed9ab4b6d4cd89: No bootstrap required, opened a new log
I20260804 14:53:41.296168 5740 ts_tablet_manager.cc:1397] T c51e30c77aad40f2ad69aa98c09f8c34 P 61d01a203880481b9bed9ab4b6d4cd89: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260804 14:53:41.297657 5740 raft_consensus.cc:348] T c51e30c77aad40f2ad69aa98c09f8c34 P 61d01a203880481b9bed9ab4b6d4cd89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.297837 5740 raft_consensus.cc:729] T c51e30c77aad40f2ad69aa98c09f8c34 P 61d01a203880481b9bed9ab4b6d4cd89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61d01a203880481b9bed9ab4b6d4cd89, State: Initialized, Role: FOLLOWER
I20260804 14:53:41.297971 5740 consensus_queue.cc:260] T c51e30c77aad40f2ad69aa98c09f8c34 P 61d01a203880481b9bed9ab4b6d4cd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.298393 5740 ts_tablet_manager.cc:1428] T c51e30c77aad40f2ad69aa98c09f8c34 P 61d01a203880481b9bed9ab4b6d4cd89: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260804 14:53:41.298597 5740 tablet_bootstrap.cc:492] T 73efc26c877b46828c2b8ccadceaa91a P 61d01a203880481b9bed9ab4b6d4cd89: Bootstrap starting.
I20260804 14:53:41.298964 5773 tablet_bootstrap.cc:492] T be8a881726c9437283de67de4c31559a P 2fb91ff838c74c0490ce3abaca0e933b: Bootstrap starting.
I20260804 14:53:41.299780 5740 tablet_bootstrap.cc:654] T 73efc26c877b46828c2b8ccadceaa91a P 61d01a203880481b9bed9ab4b6d4cd89: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:41.301600 5740 tablet_bootstrap.cc:492] T 73efc26c877b46828c2b8ccadceaa91a P 61d01a203880481b9bed9ab4b6d4cd89: No bootstrap required, opened a new log
I20260804 14:53:41.301703 5740 ts_tablet_manager.cc:1397] T 73efc26c877b46828c2b8ccadceaa91a P 61d01a203880481b9bed9ab4b6d4cd89: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260804 14:53:41.301695 5773 tablet_bootstrap.cc:654] T be8a881726c9437283de67de4c31559a P 2fb91ff838c74c0490ce3abaca0e933b: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:41.302222 5740 raft_consensus.cc:348] T 73efc26c877b46828c2b8ccadceaa91a P 61d01a203880481b9bed9ab4b6d4cd89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.302352 5740 raft_consensus.cc:729] T 73efc26c877b46828c2b8ccadceaa91a P 61d01a203880481b9bed9ab4b6d4cd89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61d01a203880481b9bed9ab4b6d4cd89, State: Initialized, Role: FOLLOWER
I20260804 14:53:41.302456 5740 consensus_queue.cc:260] T 73efc26c877b46828c2b8ccadceaa91a P 61d01a203880481b9bed9ab4b6d4cd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.302635 5773 log.cc:824] T be8a881726c9437283de67de4c31559a P 2fb91ff838c74c0490ce3abaca0e933b: Log is configured to *not* fsync() on all Append() calls
I20260804 14:53:41.302726 5740 ts_tablet_manager.cc:1428] T 73efc26c877b46828c2b8ccadceaa91a P 61d01a203880481b9bed9ab4b6d4cd89: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260804 14:53:41.302891 5740 tablet_bootstrap.cc:492] T dea55fb16e734405a55313a86ac8e378 P 61d01a203880481b9bed9ab4b6d4cd89: Bootstrap starting.
I20260804 14:53:41.303918 5740 tablet_bootstrap.cc:654] T dea55fb16e734405a55313a86ac8e378 P 61d01a203880481b9bed9ab4b6d4cd89: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:41.306147 5740 tablet_bootstrap.cc:492] T dea55fb16e734405a55313a86ac8e378 P 61d01a203880481b9bed9ab4b6d4cd89: No bootstrap required, opened a new log
I20260804 14:53:41.306252 5740 ts_tablet_manager.cc:1397] T dea55fb16e734405a55313a86ac8e378 P 61d01a203880481b9bed9ab4b6d4cd89: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260804 14:53:41.306886 5740 raft_consensus.cc:348] T dea55fb16e734405a55313a86ac8e378 P 61d01a203880481b9bed9ab4b6d4cd89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.306990 5740 raft_consensus.cc:729] T dea55fb16e734405a55313a86ac8e378 P 61d01a203880481b9bed9ab4b6d4cd89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61d01a203880481b9bed9ab4b6d4cd89, State: Initialized, Role: FOLLOWER
I20260804 14:53:41.307070 5740 consensus_queue.cc:260] T dea55fb16e734405a55313a86ac8e378 P 61d01a203880481b9bed9ab4b6d4cd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.307302 5773 tablet_bootstrap.cc:492] T be8a881726c9437283de67de4c31559a P 2fb91ff838c74c0490ce3abaca0e933b: No bootstrap required, opened a new log
I20260804 14:53:41.307354 5740 ts_tablet_manager.cc:1428] T dea55fb16e734405a55313a86ac8e378 P 61d01a203880481b9bed9ab4b6d4cd89: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260804 14:53:41.307524 5773 ts_tablet_manager.cc:1397] T be8a881726c9437283de67de4c31559a P 2fb91ff838c74c0490ce3abaca0e933b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260804 14:53:41.307646 5740 tablet_bootstrap.cc:492] T dd390684aa114e3b97056bf52dab4de1 P 61d01a203880481b9bed9ab4b6d4cd89: Bootstrap starting.
I20260804 14:53:41.308741 5740 tablet_bootstrap.cc:654] T dd390684aa114e3b97056bf52dab4de1 P 61d01a203880481b9bed9ab4b6d4cd89: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:41.310045 5740 tablet_bootstrap.cc:492] T dd390684aa114e3b97056bf52dab4de1 P 61d01a203880481b9bed9ab4b6d4cd89: No bootstrap required, opened a new log
I20260804 14:53:41.310215 5740 ts_tablet_manager.cc:1397] T dd390684aa114e3b97056bf52dab4de1 P 61d01a203880481b9bed9ab4b6d4cd89: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260804 14:53:41.310679 5740 raft_consensus.cc:348] T dd390684aa114e3b97056bf52dab4de1 P 61d01a203880481b9bed9ab4b6d4cd89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.310808 5740 raft_consensus.cc:729] T dd390684aa114e3b97056bf52dab4de1 P 61d01a203880481b9bed9ab4b6d4cd89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61d01a203880481b9bed9ab4b6d4cd89, State: Initialized, Role: FOLLOWER
I20260804 14:53:41.310911 5740 consensus_queue.cc:260] T dd390684aa114e3b97056bf52dab4de1 P 61d01a203880481b9bed9ab4b6d4cd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.311240 5740 ts_tablet_manager.cc:1428] T dd390684aa114e3b97056bf52dab4de1 P 61d01a203880481b9bed9ab4b6d4cd89: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260804 14:53:41.312258 5773 raft_consensus.cc:348] T be8a881726c9437283de67de4c31559a P 2fb91ff838c74c0490ce3abaca0e933b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.312590 5773 raft_consensus.cc:729] T be8a881726c9437283de67de4c31559a P 2fb91ff838c74c0490ce3abaca0e933b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb91ff838c74c0490ce3abaca0e933b, State: Initialized, Role: FOLLOWER
I20260804 14:53:41.312983 5773 consensus_queue.cc:260] T be8a881726c9437283de67de4c31559a P 2fb91ff838c74c0490ce3abaca0e933b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.313897 5346 heartbeater.cc:503] Master 127.24.6.190:37391 was elected leader, sending a full tablet report...
I20260804 14:53:41.314282 5773 ts_tablet_manager.cc:1428] T be8a881726c9437283de67de4c31559a P 2fb91ff838c74c0490ce3abaca0e933b: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260804 14:53:41.314610 5773 tablet_bootstrap.cc:492] T c51e30c77aad40f2ad69aa98c09f8c34 P 2fb91ff838c74c0490ce3abaca0e933b: Bootstrap starting.
I20260804 14:53:41.315904 5773 tablet_bootstrap.cc:654] T c51e30c77aad40f2ad69aa98c09f8c34 P 2fb91ff838c74c0490ce3abaca0e933b: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:41.317595 5773 tablet_bootstrap.cc:492] T c51e30c77aad40f2ad69aa98c09f8c34 P 2fb91ff838c74c0490ce3abaca0e933b: No bootstrap required, opened a new log
I20260804 14:53:41.317736 5773 ts_tablet_manager.cc:1397] T c51e30c77aad40f2ad69aa98c09f8c34 P 2fb91ff838c74c0490ce3abaca0e933b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260804 14:53:41.318611 5773 raft_consensus.cc:348] T c51e30c77aad40f2ad69aa98c09f8c34 P 2fb91ff838c74c0490ce3abaca0e933b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.318794 5773 raft_consensus.cc:729] T c51e30c77aad40f2ad69aa98c09f8c34 P 2fb91ff838c74c0490ce3abaca0e933b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb91ff838c74c0490ce3abaca0e933b, State: Initialized, Role: FOLLOWER
I20260804 14:53:41.318918 5773 consensus_queue.cc:260] T c51e30c77aad40f2ad69aa98c09f8c34 P 2fb91ff838c74c0490ce3abaca0e933b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.319293 5773 ts_tablet_manager.cc:1428] T c51e30c77aad40f2ad69aa98c09f8c34 P 2fb91ff838c74c0490ce3abaca0e933b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260804 14:53:41.319510 5773 tablet_bootstrap.cc:492] T 5dcdea9fba5f4e908350c27a454d97ca P 2fb91ff838c74c0490ce3abaca0e933b: Bootstrap starting.
I20260804 14:53:41.320580 5773 tablet_bootstrap.cc:654] T 5dcdea9fba5f4e908350c27a454d97ca P 2fb91ff838c74c0490ce3abaca0e933b: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:41.321925 5773 tablet_bootstrap.cc:492] T 5dcdea9fba5f4e908350c27a454d97ca P 2fb91ff838c74c0490ce3abaca0e933b: No bootstrap required, opened a new log
I20260804 14:53:41.322005 5773 ts_tablet_manager.cc:1397] T 5dcdea9fba5f4e908350c27a454d97ca P 2fb91ff838c74c0490ce3abaca0e933b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260804 14:53:41.322496 5773 raft_consensus.cc:348] T 5dcdea9fba5f4e908350c27a454d97ca P 2fb91ff838c74c0490ce3abaca0e933b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.322650 5773 raft_consensus.cc:729] T 5dcdea9fba5f4e908350c27a454d97ca P 2fb91ff838c74c0490ce3abaca0e933b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb91ff838c74c0490ce3abaca0e933b, State: Initialized, Role: FOLLOWER
I20260804 14:53:41.322806 5773 consensus_queue.cc:260] T 5dcdea9fba5f4e908350c27a454d97ca P 2fb91ff838c74c0490ce3abaca0e933b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.323096 5773 ts_tablet_manager.cc:1428] T 5dcdea9fba5f4e908350c27a454d97ca P 2fb91ff838c74c0490ce3abaca0e933b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260804 14:53:41.323318 5773 tablet_bootstrap.cc:492] T 73efc26c877b46828c2b8ccadceaa91a P 2fb91ff838c74c0490ce3abaca0e933b: Bootstrap starting.
I20260804 14:53:41.324623 5773 tablet_bootstrap.cc:654] T 73efc26c877b46828c2b8ccadceaa91a P 2fb91ff838c74c0490ce3abaca0e933b: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:41.325799 5773 tablet_bootstrap.cc:492] T 73efc26c877b46828c2b8ccadceaa91a P 2fb91ff838c74c0490ce3abaca0e933b: No bootstrap required, opened a new log
I20260804 14:53:41.325908 5773 ts_tablet_manager.cc:1397] T 73efc26c877b46828c2b8ccadceaa91a P 2fb91ff838c74c0490ce3abaca0e933b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260804 14:53:41.326357 5773 raft_consensus.cc:348] T 73efc26c877b46828c2b8ccadceaa91a P 2fb91ff838c74c0490ce3abaca0e933b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.326476 5773 raft_consensus.cc:729] T 73efc26c877b46828c2b8ccadceaa91a P 2fb91ff838c74c0490ce3abaca0e933b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb91ff838c74c0490ce3abaca0e933b, State: Initialized, Role: FOLLOWER
I20260804 14:53:41.326596 5773 consensus_queue.cc:260] T 73efc26c877b46828c2b8ccadceaa91a P 2fb91ff838c74c0490ce3abaca0e933b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.326939 5773 ts_tablet_manager.cc:1428] T 73efc26c877b46828c2b8ccadceaa91a P 2fb91ff838c74c0490ce3abaca0e933b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260804 14:53:41.327137 5773 tablet_bootstrap.cc:492] T 57ee2bd239614464a99304a04ddc67bf P 2fb91ff838c74c0490ce3abaca0e933b: Bootstrap starting.
I20260804 14:53:41.328094 5773 tablet_bootstrap.cc:654] T 57ee2bd239614464a99304a04ddc67bf P 2fb91ff838c74c0490ce3abaca0e933b: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:41.330250 5773 tablet_bootstrap.cc:492] T 57ee2bd239614464a99304a04ddc67bf P 2fb91ff838c74c0490ce3abaca0e933b: No bootstrap required, opened a new log
I20260804 14:53:41.330358 5773 ts_tablet_manager.cc:1397] T 57ee2bd239614464a99304a04ddc67bf P 2fb91ff838c74c0490ce3abaca0e933b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260804 14:53:41.330834 5773 raft_consensus.cc:348] T 57ee2bd239614464a99304a04ddc67bf P 2fb91ff838c74c0490ce3abaca0e933b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.330984 5773 raft_consensus.cc:729] T 57ee2bd239614464a99304a04ddc67bf P 2fb91ff838c74c0490ce3abaca0e933b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb91ff838c74c0490ce3abaca0e933b, State: Initialized, Role: FOLLOWER
I20260804 14:53:41.331082 5773 consensus_queue.cc:260] T 57ee2bd239614464a99304a04ddc67bf P 2fb91ff838c74c0490ce3abaca0e933b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.331408 5773 ts_tablet_manager.cc:1428] T 57ee2bd239614464a99304a04ddc67bf P 2fb91ff838c74c0490ce3abaca0e933b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260804 14:53:41.331634 5773 tablet_bootstrap.cc:492] T 335ce7f1cab04f2d8c23b89e30fca53d P 2fb91ff838c74c0490ce3abaca0e933b: Bootstrap starting.
I20260804 14:53:41.332845 5773 tablet_bootstrap.cc:654] T 335ce7f1cab04f2d8c23b89e30fca53d P 2fb91ff838c74c0490ce3abaca0e933b: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:41.334333 5773 tablet_bootstrap.cc:492] T 335ce7f1cab04f2d8c23b89e30fca53d P 2fb91ff838c74c0490ce3abaca0e933b: No bootstrap required, opened a new log
I20260804 14:53:41.334479 5773 ts_tablet_manager.cc:1397] T 335ce7f1cab04f2d8c23b89e30fca53d P 2fb91ff838c74c0490ce3abaca0e933b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260804 14:53:41.334969 5773 raft_consensus.cc:348] T 335ce7f1cab04f2d8c23b89e30fca53d P 2fb91ff838c74c0490ce3abaca0e933b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.335110 5773 raft_consensus.cc:729] T 335ce7f1cab04f2d8c23b89e30fca53d P 2fb91ff838c74c0490ce3abaca0e933b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb91ff838c74c0490ce3abaca0e933b, State: Initialized, Role: FOLLOWER
I20260804 14:53:41.335263 5773 consensus_queue.cc:260] T 335ce7f1cab04f2d8c23b89e30fca53d P 2fb91ff838c74c0490ce3abaca0e933b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.335628 5773 ts_tablet_manager.cc:1428] T 335ce7f1cab04f2d8c23b89e30fca53d P 2fb91ff838c74c0490ce3abaca0e933b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260804 14:53:41.335848 5773 tablet_bootstrap.cc:492] T 9eda8efcee174a59a84c84e9900b1a85 P 2fb91ff838c74c0490ce3abaca0e933b: Bootstrap starting.
I20260804 14:53:41.337100 5773 tablet_bootstrap.cc:654] T 9eda8efcee174a59a84c84e9900b1a85 P 2fb91ff838c74c0490ce3abaca0e933b: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:41.338414 5773 tablet_bootstrap.cc:492] T 9eda8efcee174a59a84c84e9900b1a85 P 2fb91ff838c74c0490ce3abaca0e933b: No bootstrap required, opened a new log
I20260804 14:53:41.338569 5773 ts_tablet_manager.cc:1397] T 9eda8efcee174a59a84c84e9900b1a85 P 2fb91ff838c74c0490ce3abaca0e933b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260804 14:53:41.339002 5773 raft_consensus.cc:348] T 9eda8efcee174a59a84c84e9900b1a85 P 2fb91ff838c74c0490ce3abaca0e933b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.339145 5773 raft_consensus.cc:729] T 9eda8efcee174a59a84c84e9900b1a85 P 2fb91ff838c74c0490ce3abaca0e933b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb91ff838c74c0490ce3abaca0e933b, State: Initialized, Role: FOLLOWER
I20260804 14:53:41.339285 5773 consensus_queue.cc:260] T 9eda8efcee174a59a84c84e9900b1a85 P 2fb91ff838c74c0490ce3abaca0e933b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.339543 5773 ts_tablet_manager.cc:1428] T 9eda8efcee174a59a84c84e9900b1a85 P 2fb91ff838c74c0490ce3abaca0e933b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260804 14:53:41.339777 5773 tablet_bootstrap.cc:492] T 7227e11e83004ac7a3079cb8e7a860f5 P 2fb91ff838c74c0490ce3abaca0e933b: Bootstrap starting.
I20260804 14:53:41.341713 5773 tablet_bootstrap.cc:654] T 7227e11e83004ac7a3079cb8e7a860f5 P 2fb91ff838c74c0490ce3abaca0e933b: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:41.343699 5773 tablet_bootstrap.cc:492] T 7227e11e83004ac7a3079cb8e7a860f5 P 2fb91ff838c74c0490ce3abaca0e933b: No bootstrap required, opened a new log
I20260804 14:53:41.343824 5773 ts_tablet_manager.cc:1397] T 7227e11e83004ac7a3079cb8e7a860f5 P 2fb91ff838c74c0490ce3abaca0e933b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260804 14:53:41.344559 5773 raft_consensus.cc:348] T 7227e11e83004ac7a3079cb8e7a860f5 P 2fb91ff838c74c0490ce3abaca0e933b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.344732 5773 raft_consensus.cc:729] T 7227e11e83004ac7a3079cb8e7a860f5 P 2fb91ff838c74c0490ce3abaca0e933b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb91ff838c74c0490ce3abaca0e933b, State: Initialized, Role: FOLLOWER
I20260804 14:53:41.344877 5773 consensus_queue.cc:260] T 7227e11e83004ac7a3079cb8e7a860f5 P 2fb91ff838c74c0490ce3abaca0e933b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.345260 5773 ts_tablet_manager.cc:1428] T 7227e11e83004ac7a3079cb8e7a860f5 P 2fb91ff838c74c0490ce3abaca0e933b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260804 14:53:41.345480 5773 tablet_bootstrap.cc:492] T dd390684aa114e3b97056bf52dab4de1 P 2fb91ff838c74c0490ce3abaca0e933b: Bootstrap starting.
I20260804 14:53:41.346436 5773 tablet_bootstrap.cc:654] T dd390684aa114e3b97056bf52dab4de1 P 2fb91ff838c74c0490ce3abaca0e933b: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:41.347651 5773 tablet_bootstrap.cc:492] T dd390684aa114e3b97056bf52dab4de1 P 2fb91ff838c74c0490ce3abaca0e933b: No bootstrap required, opened a new log
I20260804 14:53:41.347764 5773 ts_tablet_manager.cc:1397] T dd390684aa114e3b97056bf52dab4de1 P 2fb91ff838c74c0490ce3abaca0e933b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260804 14:53:41.348258 5773 raft_consensus.cc:348] T dd390684aa114e3b97056bf52dab4de1 P 2fb91ff838c74c0490ce3abaca0e933b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.348388 5773 raft_consensus.cc:729] T dd390684aa114e3b97056bf52dab4de1 P 2fb91ff838c74c0490ce3abaca0e933b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb91ff838c74c0490ce3abaca0e933b, State: Initialized, Role: FOLLOWER
I20260804 14:53:41.348501 5773 consensus_queue.cc:260] T dd390684aa114e3b97056bf52dab4de1 P 2fb91ff838c74c0490ce3abaca0e933b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.348794 5773 ts_tablet_manager.cc:1428] T dd390684aa114e3b97056bf52dab4de1 P 2fb91ff838c74c0490ce3abaca0e933b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260804 14:53:41.349072 5773 tablet_bootstrap.cc:492] T dea55fb16e734405a55313a86ac8e378 P 2fb91ff838c74c0490ce3abaca0e933b: Bootstrap starting.
I20260804 14:53:41.350318 5773 tablet_bootstrap.cc:654] T dea55fb16e734405a55313a86ac8e378 P 2fb91ff838c74c0490ce3abaca0e933b: Neither blocks nor log segments found. Creating new log.
I20260804 14:53:41.351641 5773 tablet_bootstrap.cc:492] T dea55fb16e734405a55313a86ac8e378 P 2fb91ff838c74c0490ce3abaca0e933b: No bootstrap required, opened a new log
I20260804 14:53:41.351758 5773 ts_tablet_manager.cc:1397] T dea55fb16e734405a55313a86ac8e378 P 2fb91ff838c74c0490ce3abaca0e933b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260804 14:53:41.352205 5773 raft_consensus.cc:348] T dea55fb16e734405a55313a86ac8e378 P 2fb91ff838c74c0490ce3abaca0e933b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.352361 5773 raft_consensus.cc:729] T dea55fb16e734405a55313a86ac8e378 P 2fb91ff838c74c0490ce3abaca0e933b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb91ff838c74c0490ce3abaca0e933b, State: Initialized, Role: FOLLOWER
I20260804 14:53:41.352496 5773 consensus_queue.cc:260] T dea55fb16e734405a55313a86ac8e378 P 2fb91ff838c74c0490ce3abaca0e933b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.352880 5773 ts_tablet_manager.cc:1428] T dea55fb16e734405a55313a86ac8e378 P 2fb91ff838c74c0490ce3abaca0e933b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260804 14:53:41.380745 5463 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dea55fb16e734405a55313a86ac8e378"
dest_uuid: "3e653e9eb883427686d59e1134c442e1"
from {username='slave'} at 127.0.0.1:43062
I20260804 14:53:41.381033 5463 raft_consensus.cc:482] T dea55fb16e734405a55313a86ac8e378 P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260804 14:53:41.381145 5463 raft_consensus.cc:3037] T dea55fb16e734405a55313a86ac8e378 P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:41.382270 5463 raft_consensus.cc:504] T dea55fb16e734405a55313a86ac8e378 P 3e653e9eb883427686d59e1134c442e1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.383419 5463 leader_election.cc:288] T dea55fb16e734405a55313a86ac8e378 P 3e653e9eb883427686d59e1134c442e1 [CANDIDATE]: Term 1 election: Requested vote from peers 2fb91ff838c74c0490ce3abaca0e933b (127.24.6.129:44579), 61d01a203880481b9bed9ab4b6d4cd89 (127.24.6.131:44621)
I20260804 14:53:41.389122 5641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea55fb16e734405a55313a86ac8e378" candidate_uuid: "3e653e9eb883427686d59e1134c442e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "61d01a203880481b9bed9ab4b6d4cd89"
I20260804 14:53:41.389281 5283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea55fb16e734405a55313a86ac8e378" candidate_uuid: "3e653e9eb883427686d59e1134c442e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2fb91ff838c74c0490ce3abaca0e933b"
I20260804 14:53:41.389508 5641 raft_consensus.cc:3037] T dea55fb16e734405a55313a86ac8e378 P 61d01a203880481b9bed9ab4b6d4cd89 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:41.389591 5283 raft_consensus.cc:3037] T dea55fb16e734405a55313a86ac8e378 P 2fb91ff838c74c0490ce3abaca0e933b [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:41.390952 5283 raft_consensus.cc:2454] T dea55fb16e734405a55313a86ac8e378 P 2fb91ff838c74c0490ce3abaca0e933b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e653e9eb883427686d59e1134c442e1 in term 1.
I20260804 14:53:41.390956 5641 raft_consensus.cc:2454] T dea55fb16e734405a55313a86ac8e378 P 61d01a203880481b9bed9ab4b6d4cd89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e653e9eb883427686d59e1134c442e1 in term 1.
I20260804 14:53:41.391476 5371 leader_election.cc:302] T dea55fb16e734405a55313a86ac8e378 P 3e653e9eb883427686d59e1134c442e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e653e9eb883427686d59e1134c442e1, 61d01a203880481b9bed9ab4b6d4cd89; no voters:
I20260804 14:53:41.391853 5748 raft_consensus.cc:2781] T dea55fb16e734405a55313a86ac8e378 P 3e653e9eb883427686d59e1134c442e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:41.391981 5748 raft_consensus.cc:686] T dea55fb16e734405a55313a86ac8e378 P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: Becoming Leader. State: Replica: 3e653e9eb883427686d59e1134c442e1, State: Running, Role: LEADER
I20260804 14:53:41.392414 5748 consensus_queue.cc:237] T dea55fb16e734405a55313a86ac8e378 P 3e653e9eb883427686d59e1134c442e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.396113 5090 catalog_manager.cc:5495] T dea55fb16e734405a55313a86ac8e378 P 3e653e9eb883427686d59e1134c442e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e653e9eb883427686d59e1134c442e1 (127.24.6.130). New cstate: current_term: 1 leader_uuid: "3e653e9eb883427686d59e1134c442e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } health_report { overall_health: UNKNOWN } } }
I20260804 14:53:41.402305 5463 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dd390684aa114e3b97056bf52dab4de1"
dest_uuid: "3e653e9eb883427686d59e1134c442e1"
from {username='slave'} at 127.0.0.1:43062
I20260804 14:53:41.402493 5463 raft_consensus.cc:482] T dd390684aa114e3b97056bf52dab4de1 P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260804 14:53:41.402544 5463 raft_consensus.cc:3037] T dd390684aa114e3b97056bf52dab4de1 P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:41.403690 5463 raft_consensus.cc:504] T dd390684aa114e3b97056bf52dab4de1 P 3e653e9eb883427686d59e1134c442e1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.404067 5463 leader_election.cc:288] T dd390684aa114e3b97056bf52dab4de1 P 3e653e9eb883427686d59e1134c442e1 [CANDIDATE]: Term 1 election: Requested vote from peers 2fb91ff838c74c0490ce3abaca0e933b (127.24.6.129:44579), 61d01a203880481b9bed9ab4b6d4cd89 (127.24.6.131:44621)
I20260804 14:53:41.404565 5283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd390684aa114e3b97056bf52dab4de1" candidate_uuid: "3e653e9eb883427686d59e1134c442e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2fb91ff838c74c0490ce3abaca0e933b"
I20260804 14:53:41.404650 5641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd390684aa114e3b97056bf52dab4de1" candidate_uuid: "3e653e9eb883427686d59e1134c442e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "61d01a203880481b9bed9ab4b6d4cd89"
I20260804 14:53:41.404724 5283 raft_consensus.cc:3037] T dd390684aa114e3b97056bf52dab4de1 P 2fb91ff838c74c0490ce3abaca0e933b [term 0 FOLLOWER]: Advancing to term 1
W20260804 14:53:41.404726 5711 tablet.cc:2369] T 335ce7f1cab04f2d8c23b89e30fca53d P 61d01a203880481b9bed9ab4b6d4cd89: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260804 14:53:41.404793 5641 raft_consensus.cc:3037] T dd390684aa114e3b97056bf52dab4de1 P 61d01a203880481b9bed9ab4b6d4cd89 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:41.405821 5283 raft_consensus.cc:2454] T dd390684aa114e3b97056bf52dab4de1 P 2fb91ff838c74c0490ce3abaca0e933b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e653e9eb883427686d59e1134c442e1 in term 1.
I20260804 14:53:41.405871 5641 raft_consensus.cc:2454] T dd390684aa114e3b97056bf52dab4de1 P 61d01a203880481b9bed9ab4b6d4cd89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e653e9eb883427686d59e1134c442e1 in term 1.
I20260804 14:53:41.406177 5371 leader_election.cc:302] T dd390684aa114e3b97056bf52dab4de1 P 3e653e9eb883427686d59e1134c442e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e653e9eb883427686d59e1134c442e1, 61d01a203880481b9bed9ab4b6d4cd89; no voters:
I20260804 14:53:41.406402 5748 raft_consensus.cc:2781] T dd390684aa114e3b97056bf52dab4de1 P 3e653e9eb883427686d59e1134c442e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:41.406515 5748 raft_consensus.cc:686] T dd390684aa114e3b97056bf52dab4de1 P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: Becoming Leader. State: Replica: 3e653e9eb883427686d59e1134c442e1, State: Running, Role: LEADER
I20260804 14:53:41.406678 5748 consensus_queue.cc:237] T dd390684aa114e3b97056bf52dab4de1 P 3e653e9eb883427686d59e1134c442e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.408772 5090 catalog_manager.cc:5495] T dd390684aa114e3b97056bf52dab4de1 P 3e653e9eb883427686d59e1134c442e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e653e9eb883427686d59e1134c442e1 (127.24.6.130). New cstate: current_term: 1 leader_uuid: "3e653e9eb883427686d59e1134c442e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } health_report { overall_health: UNKNOWN } } }
I20260804 14:53:41.414724 5463 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7227e11e83004ac7a3079cb8e7a860f5"
dest_uuid: "3e653e9eb883427686d59e1134c442e1"
from {username='slave'} at 127.0.0.1:43062
I20260804 14:53:41.414894 5463 raft_consensus.cc:482] T 7227e11e83004ac7a3079cb8e7a860f5 P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260804 14:53:41.414947 5463 raft_consensus.cc:3037] T 7227e11e83004ac7a3079cb8e7a860f5 P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:41.415911 5463 raft_consensus.cc:504] T 7227e11e83004ac7a3079cb8e7a860f5 P 3e653e9eb883427686d59e1134c442e1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.416323 5463 leader_election.cc:288] T 7227e11e83004ac7a3079cb8e7a860f5 P 3e653e9eb883427686d59e1134c442e1 [CANDIDATE]: Term 1 election: Requested vote from peers 2fb91ff838c74c0490ce3abaca0e933b (127.24.6.129:44579), 61d01a203880481b9bed9ab4b6d4cd89 (127.24.6.131:44621)
I20260804 14:53:41.416906 5283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7227e11e83004ac7a3079cb8e7a860f5" candidate_uuid: "3e653e9eb883427686d59e1134c442e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2fb91ff838c74c0490ce3abaca0e933b"
I20260804 14:53:41.416944 5641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7227e11e83004ac7a3079cb8e7a860f5" candidate_uuid: "3e653e9eb883427686d59e1134c442e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "61d01a203880481b9bed9ab4b6d4cd89"
I20260804 14:53:41.417104 5283 raft_consensus.cc:3037] T 7227e11e83004ac7a3079cb8e7a860f5 P 2fb91ff838c74c0490ce3abaca0e933b [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:41.417140 5641 raft_consensus.cc:3037] T 7227e11e83004ac7a3079cb8e7a860f5 P 61d01a203880481b9bed9ab4b6d4cd89 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:41.418303 5641 raft_consensus.cc:2454] T 7227e11e83004ac7a3079cb8e7a860f5 P 61d01a203880481b9bed9ab4b6d4cd89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e653e9eb883427686d59e1134c442e1 in term 1.
I20260804 14:53:41.418308 5283 raft_consensus.cc:2454] T 7227e11e83004ac7a3079cb8e7a860f5 P 2fb91ff838c74c0490ce3abaca0e933b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e653e9eb883427686d59e1134c442e1 in term 1.
I20260804 14:53:41.418668 5371 leader_election.cc:302] T 7227e11e83004ac7a3079cb8e7a860f5 P 3e653e9eb883427686d59e1134c442e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb91ff838c74c0490ce3abaca0e933b, 3e653e9eb883427686d59e1134c442e1; no voters:
I20260804 14:53:41.418875 5748 raft_consensus.cc:2781] T 7227e11e83004ac7a3079cb8e7a860f5 P 3e653e9eb883427686d59e1134c442e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:41.418972 5748 raft_consensus.cc:686] T 7227e11e83004ac7a3079cb8e7a860f5 P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: Becoming Leader. State: Replica: 3e653e9eb883427686d59e1134c442e1, State: Running, Role: LEADER
I20260804 14:53:41.419142 5748 consensus_queue.cc:237] T 7227e11e83004ac7a3079cb8e7a860f5 P 3e653e9eb883427686d59e1134c442e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.421298 5090 catalog_manager.cc:5495] T 7227e11e83004ac7a3079cb8e7a860f5 P 3e653e9eb883427686d59e1134c442e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e653e9eb883427686d59e1134c442e1 (127.24.6.130). New cstate: current_term: 1 leader_uuid: "3e653e9eb883427686d59e1134c442e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } health_report { overall_health: UNKNOWN } } }
I20260804 14:53:41.426590 5463 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9eda8efcee174a59a84c84e9900b1a85"
dest_uuid: "3e653e9eb883427686d59e1134c442e1"
from {username='slave'} at 127.0.0.1:43062
I20260804 14:53:41.426800 5463 raft_consensus.cc:482] T 9eda8efcee174a59a84c84e9900b1a85 P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260804 14:53:41.426873 5463 raft_consensus.cc:3037] T 9eda8efcee174a59a84c84e9900b1a85 P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:41.428086 5463 raft_consensus.cc:504] T 9eda8efcee174a59a84c84e9900b1a85 P 3e653e9eb883427686d59e1134c442e1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.428579 5463 leader_election.cc:288] T 9eda8efcee174a59a84c84e9900b1a85 P 3e653e9eb883427686d59e1134c442e1 [CANDIDATE]: Term 1 election: Requested vote from peers 2fb91ff838c74c0490ce3abaca0e933b (127.24.6.129:44579), 61d01a203880481b9bed9ab4b6d4cd89 (127.24.6.131:44621)
I20260804 14:53:41.429235 5283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eda8efcee174a59a84c84e9900b1a85" candidate_uuid: "3e653e9eb883427686d59e1134c442e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2fb91ff838c74c0490ce3abaca0e933b"
I20260804 14:53:41.429280 5641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eda8efcee174a59a84c84e9900b1a85" candidate_uuid: "3e653e9eb883427686d59e1134c442e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "61d01a203880481b9bed9ab4b6d4cd89"
I20260804 14:53:41.429386 5283 raft_consensus.cc:3037] T 9eda8efcee174a59a84c84e9900b1a85 P 2fb91ff838c74c0490ce3abaca0e933b [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:41.429440 5641 raft_consensus.cc:3037] T 9eda8efcee174a59a84c84e9900b1a85 P 61d01a203880481b9bed9ab4b6d4cd89 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:41.430538 5283 raft_consensus.cc:2454] T 9eda8efcee174a59a84c84e9900b1a85 P 2fb91ff838c74c0490ce3abaca0e933b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e653e9eb883427686d59e1134c442e1 in term 1.
I20260804 14:53:41.430651 5641 raft_consensus.cc:2454] T 9eda8efcee174a59a84c84e9900b1a85 P 61d01a203880481b9bed9ab4b6d4cd89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e653e9eb883427686d59e1134c442e1 in term 1.
I20260804 14:53:41.430946 5371 leader_election.cc:302] T 9eda8efcee174a59a84c84e9900b1a85 P 3e653e9eb883427686d59e1134c442e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb91ff838c74c0490ce3abaca0e933b, 3e653e9eb883427686d59e1134c442e1; no voters:
I20260804 14:53:41.431246 5748 raft_consensus.cc:2781] T 9eda8efcee174a59a84c84e9900b1a85 P 3e653e9eb883427686d59e1134c442e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:41.431355 5748 raft_consensus.cc:686] T 9eda8efcee174a59a84c84e9900b1a85 P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: Becoming Leader. State: Replica: 3e653e9eb883427686d59e1134c442e1, State: Running, Role: LEADER
I20260804 14:53:41.431509 5748 consensus_queue.cc:237] T 9eda8efcee174a59a84c84e9900b1a85 P 3e653e9eb883427686d59e1134c442e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.433633 5090 catalog_manager.cc:5495] T 9eda8efcee174a59a84c84e9900b1a85 P 3e653e9eb883427686d59e1134c442e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e653e9eb883427686d59e1134c442e1 (127.24.6.130). New cstate: current_term: 1 leader_uuid: "3e653e9eb883427686d59e1134c442e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } health_report { overall_health: UNKNOWN } } }
I20260804 14:53:41.439750 5463 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c51e30c77aad40f2ad69aa98c09f8c34"
dest_uuid: "3e653e9eb883427686d59e1134c442e1"
from {username='slave'} at 127.0.0.1:43062
I20260804 14:53:41.440056 5463 raft_consensus.cc:482] T c51e30c77aad40f2ad69aa98c09f8c34 P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260804 14:53:41.440176 5463 raft_consensus.cc:3037] T c51e30c77aad40f2ad69aa98c09f8c34 P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:41.441708 5463 raft_consensus.cc:504] T c51e30c77aad40f2ad69aa98c09f8c34 P 3e653e9eb883427686d59e1134c442e1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.442175 5463 leader_election.cc:288] T c51e30c77aad40f2ad69aa98c09f8c34 P 3e653e9eb883427686d59e1134c442e1 [CANDIDATE]: Term 1 election: Requested vote from peers 2fb91ff838c74c0490ce3abaca0e933b (127.24.6.129:44579), 61d01a203880481b9bed9ab4b6d4cd89 (127.24.6.131:44621)
I20260804 14:53:41.443040 5283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c51e30c77aad40f2ad69aa98c09f8c34" candidate_uuid: "3e653e9eb883427686d59e1134c442e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2fb91ff838c74c0490ce3abaca0e933b"
I20260804 14:53:41.443107 5641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c51e30c77aad40f2ad69aa98c09f8c34" candidate_uuid: "3e653e9eb883427686d59e1134c442e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "61d01a203880481b9bed9ab4b6d4cd89"
I20260804 14:53:41.443262 5283 raft_consensus.cc:3037] T c51e30c77aad40f2ad69aa98c09f8c34 P 2fb91ff838c74c0490ce3abaca0e933b [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:41.443290 5641 raft_consensus.cc:3037] T c51e30c77aad40f2ad69aa98c09f8c34 P 61d01a203880481b9bed9ab4b6d4cd89 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:41.444324 5641 raft_consensus.cc:2454] T c51e30c77aad40f2ad69aa98c09f8c34 P 61d01a203880481b9bed9ab4b6d4cd89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e653e9eb883427686d59e1134c442e1 in term 1.
I20260804 14:53:41.444324 5283 raft_consensus.cc:2454] T c51e30c77aad40f2ad69aa98c09f8c34 P 2fb91ff838c74c0490ce3abaca0e933b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e653e9eb883427686d59e1134c442e1 in term 1.
I20260804 14:53:41.444692 5371 leader_election.cc:302] T c51e30c77aad40f2ad69aa98c09f8c34 P 3e653e9eb883427686d59e1134c442e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb91ff838c74c0490ce3abaca0e933b, 3e653e9eb883427686d59e1134c442e1; no voters:
I20260804 14:53:41.444892 5748 raft_consensus.cc:2781] T c51e30c77aad40f2ad69aa98c09f8c34 P 3e653e9eb883427686d59e1134c442e1 [term 1 FOLLOWER]: Leader election won for term 1
W20260804 14:53:41.444995 5522 tablet.cc:2369] T 335ce7f1cab04f2d8c23b89e30fca53d P 3e653e9eb883427686d59e1134c442e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260804 14:53:41.445070 5748 raft_consensus.cc:686] T c51e30c77aad40f2ad69aa98c09f8c34 P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: Becoming Leader. State: Replica: 3e653e9eb883427686d59e1134c442e1, State: Running, Role: LEADER
I20260804 14:53:41.445324 5748 consensus_queue.cc:237] T c51e30c77aad40f2ad69aa98c09f8c34 P 3e653e9eb883427686d59e1134c442e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.447185 5090 catalog_manager.cc:5495] T c51e30c77aad40f2ad69aa98c09f8c34 P 3e653e9eb883427686d59e1134c442e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e653e9eb883427686d59e1134c442e1 (127.24.6.130). New cstate: current_term: 1 leader_uuid: "3e653e9eb883427686d59e1134c442e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } health_report { overall_health: UNKNOWN } } }
I20260804 14:53:41.454391 5463 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "be8a881726c9437283de67de4c31559a"
dest_uuid: "3e653e9eb883427686d59e1134c442e1"
from {username='slave'} at 127.0.0.1:43062
I20260804 14:53:41.454632 5463 raft_consensus.cc:482] T be8a881726c9437283de67de4c31559a P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260804 14:53:41.454725 5463 raft_consensus.cc:3037] T be8a881726c9437283de67de4c31559a P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:41.455790 5463 raft_consensus.cc:504] T be8a881726c9437283de67de4c31559a P 3e653e9eb883427686d59e1134c442e1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.456221 5463 leader_election.cc:288] T be8a881726c9437283de67de4c31559a P 3e653e9eb883427686d59e1134c442e1 [CANDIDATE]: Term 1 election: Requested vote from peers 2fb91ff838c74c0490ce3abaca0e933b (127.24.6.129:44579), 61d01a203880481b9bed9ab4b6d4cd89 (127.24.6.131:44621)
I20260804 14:53:41.456873 5283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be8a881726c9437283de67de4c31559a" candidate_uuid: "3e653e9eb883427686d59e1134c442e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2fb91ff838c74c0490ce3abaca0e933b"
I20260804 14:53:41.456919 5641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be8a881726c9437283de67de4c31559a" candidate_uuid: "3e653e9eb883427686d59e1134c442e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "61d01a203880481b9bed9ab4b6d4cd89"
I20260804 14:53:41.457058 5283 raft_consensus.cc:3037] T be8a881726c9437283de67de4c31559a P 2fb91ff838c74c0490ce3abaca0e933b [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:41.457093 5641 raft_consensus.cc:3037] T be8a881726c9437283de67de4c31559a P 61d01a203880481b9bed9ab4b6d4cd89 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:41.458122 5283 raft_consensus.cc:2454] T be8a881726c9437283de67de4c31559a P 2fb91ff838c74c0490ce3abaca0e933b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e653e9eb883427686d59e1134c442e1 in term 1.
I20260804 14:53:41.458230 5641 raft_consensus.cc:2454] T be8a881726c9437283de67de4c31559a P 61d01a203880481b9bed9ab4b6d4cd89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e653e9eb883427686d59e1134c442e1 in term 1.
I20260804 14:53:41.458547 5371 leader_election.cc:302] T be8a881726c9437283de67de4c31559a P 3e653e9eb883427686d59e1134c442e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb91ff838c74c0490ce3abaca0e933b, 3e653e9eb883427686d59e1134c442e1; no voters:
I20260804 14:53:41.458766 5748 raft_consensus.cc:2781] T be8a881726c9437283de67de4c31559a P 3e653e9eb883427686d59e1134c442e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:41.458918 5748 raft_consensus.cc:686] T be8a881726c9437283de67de4c31559a P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: Becoming Leader. State: Replica: 3e653e9eb883427686d59e1134c442e1, State: Running, Role: LEADER
I20260804 14:53:41.459090 5748 consensus_queue.cc:237] T be8a881726c9437283de67de4c31559a P 3e653e9eb883427686d59e1134c442e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.461423 5090 catalog_manager.cc:5495] T be8a881726c9437283de67de4c31559a P 3e653e9eb883427686d59e1134c442e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e653e9eb883427686d59e1134c442e1 (127.24.6.130). New cstate: current_term: 1 leader_uuid: "3e653e9eb883427686d59e1134c442e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } health_report { overall_health: UNKNOWN } } }
I20260804 14:53:41.468086 5463 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5dcdea9fba5f4e908350c27a454d97ca"
dest_uuid: "3e653e9eb883427686d59e1134c442e1"
from {username='slave'} at 127.0.0.1:43062
I20260804 14:53:41.468290 5463 raft_consensus.cc:482] T 5dcdea9fba5f4e908350c27a454d97ca P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260804 14:53:41.468349 5463 raft_consensus.cc:3037] T 5dcdea9fba5f4e908350c27a454d97ca P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:41.469491 5463 raft_consensus.cc:504] T 5dcdea9fba5f4e908350c27a454d97ca P 3e653e9eb883427686d59e1134c442e1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.469905 5463 leader_election.cc:288] T 5dcdea9fba5f4e908350c27a454d97ca P 3e653e9eb883427686d59e1134c442e1 [CANDIDATE]: Term 1 election: Requested vote from peers 2fb91ff838c74c0490ce3abaca0e933b (127.24.6.129:44579), 61d01a203880481b9bed9ab4b6d4cd89 (127.24.6.131:44621)
I20260804 14:53:41.470448 5283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcdea9fba5f4e908350c27a454d97ca" candidate_uuid: "3e653e9eb883427686d59e1134c442e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2fb91ff838c74c0490ce3abaca0e933b"
I20260804 14:53:41.470510 5641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcdea9fba5f4e908350c27a454d97ca" candidate_uuid: "3e653e9eb883427686d59e1134c442e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "61d01a203880481b9bed9ab4b6d4cd89"
I20260804 14:53:41.470674 5283 raft_consensus.cc:3037] T 5dcdea9fba5f4e908350c27a454d97ca P 2fb91ff838c74c0490ce3abaca0e933b [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:41.470688 5641 raft_consensus.cc:3037] T 5dcdea9fba5f4e908350c27a454d97ca P 61d01a203880481b9bed9ab4b6d4cd89 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:41.471773 5641 raft_consensus.cc:2454] T 5dcdea9fba5f4e908350c27a454d97ca P 61d01a203880481b9bed9ab4b6d4cd89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e653e9eb883427686d59e1134c442e1 in term 1.
I20260804 14:53:41.471784 5283 raft_consensus.cc:2454] T 5dcdea9fba5f4e908350c27a454d97ca P 2fb91ff838c74c0490ce3abaca0e933b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e653e9eb883427686d59e1134c442e1 in term 1.
I20260804 14:53:41.472105 5371 leader_election.cc:302] T 5dcdea9fba5f4e908350c27a454d97ca P 3e653e9eb883427686d59e1134c442e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e653e9eb883427686d59e1134c442e1, 61d01a203880481b9bed9ab4b6d4cd89; no voters:
I20260804 14:53:41.472328 5748 raft_consensus.cc:2781] T 5dcdea9fba5f4e908350c27a454d97ca P 3e653e9eb883427686d59e1134c442e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:41.472443 5748 raft_consensus.cc:686] T 5dcdea9fba5f4e908350c27a454d97ca P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: Becoming Leader. State: Replica: 3e653e9eb883427686d59e1134c442e1, State: Running, Role: LEADER
I20260804 14:53:41.472618 5748 consensus_queue.cc:237] T 5dcdea9fba5f4e908350c27a454d97ca P 3e653e9eb883427686d59e1134c442e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.474697 5090 catalog_manager.cc:5495] T 5dcdea9fba5f4e908350c27a454d97ca P 3e653e9eb883427686d59e1134c442e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e653e9eb883427686d59e1134c442e1 (127.24.6.130). New cstate: current_term: 1 leader_uuid: "3e653e9eb883427686d59e1134c442e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } health_report { overall_health: UNKNOWN } } }
I20260804 14:53:41.482002 5463 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "73efc26c877b46828c2b8ccadceaa91a"
dest_uuid: "3e653e9eb883427686d59e1134c442e1"
from {username='slave'} at 127.0.0.1:43062
I20260804 14:53:41.482216 5463 raft_consensus.cc:482] T 73efc26c877b46828c2b8ccadceaa91a P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260804 14:53:41.482278 5463 raft_consensus.cc:3037] T 73efc26c877b46828c2b8ccadceaa91a P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:41.483350 5463 raft_consensus.cc:504] T 73efc26c877b46828c2b8ccadceaa91a P 3e653e9eb883427686d59e1134c442e1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.483779 5463 leader_election.cc:288] T 73efc26c877b46828c2b8ccadceaa91a P 3e653e9eb883427686d59e1134c442e1 [CANDIDATE]: Term 1 election: Requested vote from peers 2fb91ff838c74c0490ce3abaca0e933b (127.24.6.129:44579), 61d01a203880481b9bed9ab4b6d4cd89 (127.24.6.131:44621)
I20260804 14:53:41.484402 5283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73efc26c877b46828c2b8ccadceaa91a" candidate_uuid: "3e653e9eb883427686d59e1134c442e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2fb91ff838c74c0490ce3abaca0e933b"
I20260804 14:53:41.484443 5641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73efc26c877b46828c2b8ccadceaa91a" candidate_uuid: "3e653e9eb883427686d59e1134c442e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "61d01a203880481b9bed9ab4b6d4cd89"
I20260804 14:53:41.484571 5641 raft_consensus.cc:3037] T 73efc26c877b46828c2b8ccadceaa91a P 61d01a203880481b9bed9ab4b6d4cd89 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:41.484582 5283 raft_consensus.cc:3037] T 73efc26c877b46828c2b8ccadceaa91a P 2fb91ff838c74c0490ce3abaca0e933b [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:41.485770 5641 raft_consensus.cc:2454] T 73efc26c877b46828c2b8ccadceaa91a P 61d01a203880481b9bed9ab4b6d4cd89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e653e9eb883427686d59e1134c442e1 in term 1.
I20260804 14:53:41.485771 5283 raft_consensus.cc:2454] T 73efc26c877b46828c2b8ccadceaa91a P 2fb91ff838c74c0490ce3abaca0e933b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e653e9eb883427686d59e1134c442e1 in term 1.
W20260804 14:53:41.485785 5347 tablet.cc:2369] T 335ce7f1cab04f2d8c23b89e30fca53d P 2fb91ff838c74c0490ce3abaca0e933b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260804 14:53:41.486238 5371 leader_election.cc:302] T 73efc26c877b46828c2b8ccadceaa91a P 3e653e9eb883427686d59e1134c442e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb91ff838c74c0490ce3abaca0e933b, 3e653e9eb883427686d59e1134c442e1; no voters:
I20260804 14:53:41.486577 5748 raft_consensus.cc:2781] T 73efc26c877b46828c2b8ccadceaa91a P 3e653e9eb883427686d59e1134c442e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:41.486726 5748 raft_consensus.cc:686] T 73efc26c877b46828c2b8ccadceaa91a P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: Becoming Leader. State: Replica: 3e653e9eb883427686d59e1134c442e1, State: Running, Role: LEADER
I20260804 14:53:41.486919 5748 consensus_queue.cc:237] T 73efc26c877b46828c2b8ccadceaa91a P 3e653e9eb883427686d59e1134c442e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.489367 5090 catalog_manager.cc:5495] T 73efc26c877b46828c2b8ccadceaa91a P 3e653e9eb883427686d59e1134c442e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e653e9eb883427686d59e1134c442e1 (127.24.6.130). New cstate: current_term: 1 leader_uuid: "3e653e9eb883427686d59e1134c442e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } health_report { overall_health: UNKNOWN } } }
I20260804 14:53:41.495965 5463 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57ee2bd239614464a99304a04ddc67bf"
dest_uuid: "3e653e9eb883427686d59e1134c442e1"
from {username='slave'} at 127.0.0.1:43062
I20260804 14:53:41.496166 5463 raft_consensus.cc:482] T 57ee2bd239614464a99304a04ddc67bf P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260804 14:53:41.496224 5463 raft_consensus.cc:3037] T 57ee2bd239614464a99304a04ddc67bf P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:41.497213 5463 raft_consensus.cc:504] T 57ee2bd239614464a99304a04ddc67bf P 3e653e9eb883427686d59e1134c442e1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.497613 5463 leader_election.cc:288] T 57ee2bd239614464a99304a04ddc67bf P 3e653e9eb883427686d59e1134c442e1 [CANDIDATE]: Term 1 election: Requested vote from peers 2fb91ff838c74c0490ce3abaca0e933b (127.24.6.129:44579), 61d01a203880481b9bed9ab4b6d4cd89 (127.24.6.131:44621)
I20260804 14:53:41.498169 5283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ee2bd239614464a99304a04ddc67bf" candidate_uuid: "3e653e9eb883427686d59e1134c442e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2fb91ff838c74c0490ce3abaca0e933b"
I20260804 14:53:41.498282 5641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ee2bd239614464a99304a04ddc67bf" candidate_uuid: "3e653e9eb883427686d59e1134c442e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "61d01a203880481b9bed9ab4b6d4cd89"
I20260804 14:53:41.498335 5283 raft_consensus.cc:3037] T 57ee2bd239614464a99304a04ddc67bf P 2fb91ff838c74c0490ce3abaca0e933b [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:41.498438 5641 raft_consensus.cc:3037] T 57ee2bd239614464a99304a04ddc67bf P 61d01a203880481b9bed9ab4b6d4cd89 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:41.499585 5283 raft_consensus.cc:2454] T 57ee2bd239614464a99304a04ddc67bf P 2fb91ff838c74c0490ce3abaca0e933b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e653e9eb883427686d59e1134c442e1 in term 1.
I20260804 14:53:41.499585 5641 raft_consensus.cc:2454] T 57ee2bd239614464a99304a04ddc67bf P 61d01a203880481b9bed9ab4b6d4cd89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e653e9eb883427686d59e1134c442e1 in term 1.
I20260804 14:53:41.499953 5371 leader_election.cc:302] T 57ee2bd239614464a99304a04ddc67bf P 3e653e9eb883427686d59e1134c442e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb91ff838c74c0490ce3abaca0e933b, 3e653e9eb883427686d59e1134c442e1; no voters:
I20260804 14:53:41.500205 5748 raft_consensus.cc:2781] T 57ee2bd239614464a99304a04ddc67bf P 3e653e9eb883427686d59e1134c442e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:41.500324 5748 raft_consensus.cc:686] T 57ee2bd239614464a99304a04ddc67bf P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: Becoming Leader. State: Replica: 3e653e9eb883427686d59e1134c442e1, State: Running, Role: LEADER
I20260804 14:53:41.500504 5748 consensus_queue.cc:237] T 57ee2bd239614464a99304a04ddc67bf P 3e653e9eb883427686d59e1134c442e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.502643 5090 catalog_manager.cc:5495] T 57ee2bd239614464a99304a04ddc67bf P 3e653e9eb883427686d59e1134c442e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e653e9eb883427686d59e1134c442e1 (127.24.6.130). New cstate: current_term: 1 leader_uuid: "3e653e9eb883427686d59e1134c442e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } health_report { overall_health: UNKNOWN } } }
I20260804 14:53:41.508383 5463 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "335ce7f1cab04f2d8c23b89e30fca53d"
dest_uuid: "3e653e9eb883427686d59e1134c442e1"
from {username='slave'} at 127.0.0.1:43062
I20260804 14:53:41.508630 5463 raft_consensus.cc:482] T 335ce7f1cab04f2d8c23b89e30fca53d P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260804 14:53:41.508719 5463 raft_consensus.cc:3037] T 335ce7f1cab04f2d8c23b89e30fca53d P 3e653e9eb883427686d59e1134c442e1 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:41.509739 5463 raft_consensus.cc:504] T 335ce7f1cab04f2d8c23b89e30fca53d P 3e653e9eb883427686d59e1134c442e1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.510185 5463 leader_election.cc:288] T 335ce7f1cab04f2d8c23b89e30fca53d P 3e653e9eb883427686d59e1134c442e1 [CANDIDATE]: Term 1 election: Requested vote from peers 2fb91ff838c74c0490ce3abaca0e933b (127.24.6.129:44579), 61d01a203880481b9bed9ab4b6d4cd89 (127.24.6.131:44621)
I20260804 14:53:41.510769 5283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "335ce7f1cab04f2d8c23b89e30fca53d" candidate_uuid: "3e653e9eb883427686d59e1134c442e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2fb91ff838c74c0490ce3abaca0e933b"
I20260804 14:53:41.510931 5283 raft_consensus.cc:3037] T 335ce7f1cab04f2d8c23b89e30fca53d P 2fb91ff838c74c0490ce3abaca0e933b [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:41.510903 5641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "335ce7f1cab04f2d8c23b89e30fca53d" candidate_uuid: "3e653e9eb883427686d59e1134c442e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "61d01a203880481b9bed9ab4b6d4cd89"
I20260804 14:53:41.511072 5641 raft_consensus.cc:3037] T 335ce7f1cab04f2d8c23b89e30fca53d P 61d01a203880481b9bed9ab4b6d4cd89 [term 0 FOLLOWER]: Advancing to term 1
I20260804 14:53:41.512108 5283 raft_consensus.cc:2454] T 335ce7f1cab04f2d8c23b89e30fca53d P 2fb91ff838c74c0490ce3abaca0e933b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e653e9eb883427686d59e1134c442e1 in term 1.
I20260804 14:53:41.512279 5641 raft_consensus.cc:2454] T 335ce7f1cab04f2d8c23b89e30fca53d P 61d01a203880481b9bed9ab4b6d4cd89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e653e9eb883427686d59e1134c442e1 in term 1.
I20260804 14:53:41.512481 5371 leader_election.cc:302] T 335ce7f1cab04f2d8c23b89e30fca53d P 3e653e9eb883427686d59e1134c442e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb91ff838c74c0490ce3abaca0e933b, 3e653e9eb883427686d59e1134c442e1; no voters:
I20260804 14:53:41.512749 5748 raft_consensus.cc:2781] T 335ce7f1cab04f2d8c23b89e30fca53d P 3e653e9eb883427686d59e1134c442e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260804 14:53:41.512878 5748 raft_consensus.cc:686] T 335ce7f1cab04f2d8c23b89e30fca53d P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: Becoming Leader. State: Replica: 3e653e9eb883427686d59e1134c442e1, State: Running, Role: LEADER
I20260804 14:53:41.513065 5748 consensus_queue.cc:237] T 335ce7f1cab04f2d8c23b89e30fca53d P 3e653e9eb883427686d59e1134c442e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.515349 5090 catalog_manager.cc:5495] T 335ce7f1cab04f2d8c23b89e30fca53d P 3e653e9eb883427686d59e1134c442e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e653e9eb883427686d59e1134c442e1 (127.24.6.130). New cstate: current_term: 1 leader_uuid: "3e653e9eb883427686d59e1134c442e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } health_report { overall_health: UNKNOWN } } }
I20260804 14:53:41.522059 5426 tablet_service.cc:1514] Processing DeleteTablet for tablet dea55fb16e734405a55313a86ac8e378 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:43062
I20260804 14:53:41.522755 5811 tablet_replica.cc:330] T dea55fb16e734405a55313a86ac8e378 P 3e653e9eb883427686d59e1134c442e1: stopping tablet replica
I20260804 14:53:41.523171 5811 raft_consensus.cc:2229] T dea55fb16e734405a55313a86ac8e378 P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: Raft consensus shutting down.
I20260804 14:53:41.523641 5811 pending_rounds.cc:62] T dea55fb16e734405a55313a86ac8e378 P 3e653e9eb883427686d59e1134c442e1: Trying to abort 1 pending ops.
I20260804 14:53:41.523756 5811 pending_rounds.cc:68] T dea55fb16e734405a55313a86ac8e378 P 3e653e9eb883427686d59e1134c442e1: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7314862986826739712 op_type: NO_OP noop_request { }
I20260804 14:53:41.523907 5811 raft_consensus.cc:2866] T dea55fb16e734405a55313a86ac8e378 P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260804 14:53:41.524001 5811 raft_consensus.cc:2258] T dea55fb16e734405a55313a86ac8e378 P 3e653e9eb883427686d59e1134c442e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260804 14:53:41.525452 5811 ts_tablet_manager.cc:1905] T dea55fb16e734405a55313a86ac8e378 P 3e653e9eb883427686d59e1134c442e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260804 14:53:41.528033 5811 ts_tablet_manager.cc:1918] T dea55fb16e734405a55313a86ac8e378 P 3e653e9eb883427686d59e1134c442e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260804 14:53:41.528172 5811 log.cc:1196] T dea55fb16e734405a55313a86ac8e378 P 3e653e9eb883427686d59e1134c442e1: Deleting WAL directory at /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/wals/dea55fb16e734405a55313a86ac8e378
I20260804 14:53:41.536335 5426 tablet_service.cc:1514] Processing DeleteTablet for tablet dd390684aa114e3b97056bf52dab4de1 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:43062
I20260804 14:53:41.536626 5811 tablet_replica.cc:330] T dd390684aa114e3b97056bf52dab4de1 P 3e653e9eb883427686d59e1134c442e1: stopping tablet replica
I20260804 14:53:41.536763 5811 raft_consensus.cc:2229] T dd390684aa114e3b97056bf52dab4de1 P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: Raft consensus shutting down.
I20260804 14:53:41.536936 5811 pending_rounds.cc:62] T dd390684aa114e3b97056bf52dab4de1 P 3e653e9eb883427686d59e1134c442e1: Trying to abort 1 pending ops.
I20260804 14:53:41.536965 5811 pending_rounds.cc:68] T dd390684aa114e3b97056bf52dab4de1 P 3e653e9eb883427686d59e1134c442e1: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7314862986883489792 op_type: NO_OP noop_request { }
I20260804 14:53:41.537050 5811 raft_consensus.cc:2866] T dd390684aa114e3b97056bf52dab4de1 P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260804 14:53:41.537092 5811 raft_consensus.cc:2258] T dd390684aa114e3b97056bf52dab4de1 P 3e653e9eb883427686d59e1134c442e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260804 14:53:41.537752 5811 ts_tablet_manager.cc:1905] T dd390684aa114e3b97056bf52dab4de1 P 3e653e9eb883427686d59e1134c442e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260804 14:53:41.539809 5811 ts_tablet_manager.cc:1918] T dd390684aa114e3b97056bf52dab4de1 P 3e653e9eb883427686d59e1134c442e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260804 14:53:41.539885 5811 log.cc:1196] T dd390684aa114e3b97056bf52dab4de1 P 3e653e9eb883427686d59e1134c442e1: Deleting WAL directory at /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/wals/dd390684aa114e3b97056bf52dab4de1
I20260804 14:53:41.550408 5426 tablet_service.cc:1514] Processing DeleteTablet for tablet 7227e11e83004ac7a3079cb8e7a860f5 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:43062
I20260804 14:53:41.550761 5811 tablet_replica.cc:330] T 7227e11e83004ac7a3079cb8e7a860f5 P 3e653e9eb883427686d59e1134c442e1: stopping tablet replica
I20260804 14:53:41.550977 5811 raft_consensus.cc:2229] T 7227e11e83004ac7a3079cb8e7a860f5 P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: Raft consensus shutting down.
I20260804 14:53:41.551270 5811 pending_rounds.cc:62] T 7227e11e83004ac7a3079cb8e7a860f5 P 3e653e9eb883427686d59e1134c442e1: Trying to abort 1 pending ops.
I20260804 14:53:41.551318 5811 pending_rounds.cc:68] T 7227e11e83004ac7a3079cb8e7a860f5 P 3e653e9eb883427686d59e1134c442e1: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7314862986934870016 op_type: NO_OP noop_request { }
I20260804 14:53:41.551450 5811 raft_consensus.cc:2866] T 7227e11e83004ac7a3079cb8e7a860f5 P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260804 14:53:41.551527 5811 raft_consensus.cc:2258] T 7227e11e83004ac7a3079cb8e7a860f5 P 3e653e9eb883427686d59e1134c442e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260804 14:53:41.552296 5811 ts_tablet_manager.cc:1905] T 7227e11e83004ac7a3079cb8e7a860f5 P 3e653e9eb883427686d59e1134c442e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260804 14:53:41.554533 5811 ts_tablet_manager.cc:1918] T 7227e11e83004ac7a3079cb8e7a860f5 P 3e653e9eb883427686d59e1134c442e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260804 14:53:41.554615 5811 log.cc:1196] T 7227e11e83004ac7a3079cb8e7a860f5 P 3e653e9eb883427686d59e1134c442e1: Deleting WAL directory at /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/wals/7227e11e83004ac7a3079cb8e7a860f5
I20260804 14:53:41.561772 5426 tablet_service.cc:1514] Processing DeleteTablet for tablet 9eda8efcee174a59a84c84e9900b1a85 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:43062
I20260804 14:53:41.562187 5811 tablet_replica.cc:330] T 9eda8efcee174a59a84c84e9900b1a85 P 3e653e9eb883427686d59e1134c442e1: stopping tablet replica
I20260804 14:53:41.562381 5811 raft_consensus.cc:2229] T 9eda8efcee174a59a84c84e9900b1a85 P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: Raft consensus shutting down.
I20260804 14:53:41.562618 5811 pending_rounds.cc:62] T 9eda8efcee174a59a84c84e9900b1a85 P 3e653e9eb883427686d59e1134c442e1: Trying to abort 1 pending ops.
I20260804 14:53:41.562660 5811 pending_rounds.cc:68] T 9eda8efcee174a59a84c84e9900b1a85 P 3e653e9eb883427686d59e1134c442e1: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7314862986985259008 op_type: NO_OP noop_request { }
I20260804 14:53:41.562773 5811 raft_consensus.cc:2866] T 9eda8efcee174a59a84c84e9900b1a85 P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260804 14:53:41.562855 5811 raft_consensus.cc:2258] T 9eda8efcee174a59a84c84e9900b1a85 P 3e653e9eb883427686d59e1134c442e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260804 14:53:41.563720 5811 ts_tablet_manager.cc:1905] T 9eda8efcee174a59a84c84e9900b1a85 P 3e653e9eb883427686d59e1134c442e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260804 14:53:41.565804 5811 ts_tablet_manager.cc:1918] T 9eda8efcee174a59a84c84e9900b1a85 P 3e653e9eb883427686d59e1134c442e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260804 14:53:41.565883 5811 log.cc:1196] T 9eda8efcee174a59a84c84e9900b1a85 P 3e653e9eb883427686d59e1134c442e1: Deleting WAL directory at /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/wals/9eda8efcee174a59a84c84e9900b1a85
I20260804 14:53:41.573727 5426 tablet_service.cc:1514] Processing DeleteTablet for tablet c51e30c77aad40f2ad69aa98c09f8c34 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:43062
I20260804 14:53:41.574098 5811 tablet_replica.cc:330] T c51e30c77aad40f2ad69aa98c09f8c34 P 3e653e9eb883427686d59e1134c442e1: stopping tablet replica
I20260804 14:53:41.574301 5811 raft_consensus.cc:2229] T c51e30c77aad40f2ad69aa98c09f8c34 P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: Raft consensus shutting down.
I20260804 14:53:41.574548 5811 pending_rounds.cc:62] T c51e30c77aad40f2ad69aa98c09f8c34 P 3e653e9eb883427686d59e1134c442e1: Trying to abort 1 pending ops.
I20260804 14:53:41.574590 5811 pending_rounds.cc:68] T c51e30c77aad40f2ad69aa98c09f8c34 P 3e653e9eb883427686d59e1134c442e1: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7314862987041624064 op_type: NO_OP noop_request { }
I20260804 14:53:41.574697 5811 raft_consensus.cc:2866] T c51e30c77aad40f2ad69aa98c09f8c34 P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260804 14:53:41.574784 5811 raft_consensus.cc:2258] T c51e30c77aad40f2ad69aa98c09f8c34 P 3e653e9eb883427686d59e1134c442e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260804 14:53:41.575692 5811 ts_tablet_manager.cc:1905] T c51e30c77aad40f2ad69aa98c09f8c34 P 3e653e9eb883427686d59e1134c442e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260804 14:53:41.578131 5811 ts_tablet_manager.cc:1918] T c51e30c77aad40f2ad69aa98c09f8c34 P 3e653e9eb883427686d59e1134c442e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260804 14:53:41.578238 5811 log.cc:1196] T c51e30c77aad40f2ad69aa98c09f8c34 P 3e653e9eb883427686d59e1134c442e1: Deleting WAL directory at /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/wals/c51e30c77aad40f2ad69aa98c09f8c34
I20260804 14:53:41.586086 5463 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "be8a881726c9437283de67de4c31559a"
dest_uuid: "3e653e9eb883427686d59e1134c442e1"
from {username='slave'} at 127.0.0.1:43062
I20260804 14:53:41.586390 5463 raft_consensus.cc:580] T be8a881726c9437283de67de4c31559a P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: Received request to step down
I20260804 14:53:41.586529 5463 raft_consensus.cc:3032] T be8a881726c9437283de67de4c31559a P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: Stepping down as leader of term 1
I20260804 14:53:41.586580 5463 raft_consensus.cc:729] T be8a881726c9437283de67de4c31559a P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3e653e9eb883427686d59e1134c442e1, State: Running, Role: LEADER
I20260804 14:53:41.586786 5463 consensus_queue.cc:260] T be8a881726c9437283de67de4c31559a P 3e653e9eb883427686d59e1134c442e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.586983 5463 raft_consensus.cc:3037] T be8a881726c9437283de67de4c31559a P 3e653e9eb883427686d59e1134c442e1 [term 1 FOLLOWER]: Advancing to term 2
I20260804 14:53:41.594231 5426 tablet_service.cc:1514] Processing DeleteTablet for tablet be8a881726c9437283de67de4c31559a with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:43062
I20260804 14:53:41.594558 5811 tablet_replica.cc:330] T be8a881726c9437283de67de4c31559a P 3e653e9eb883427686d59e1134c442e1: stopping tablet replica
I20260804 14:53:41.594743 5811 raft_consensus.cc:2229] T be8a881726c9437283de67de4c31559a P 3e653e9eb883427686d59e1134c442e1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260804 14:53:41.594898 5811 pending_rounds.cc:62] T be8a881726c9437283de67de4c31559a P 3e653e9eb883427686d59e1134c442e1: Trying to abort 1 pending ops.
I20260804 14:53:41.594952 5811 pending_rounds.cc:68] T be8a881726c9437283de67de4c31559a P 3e653e9eb883427686d59e1134c442e1: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7314862987098537984 op_type: NO_OP noop_request { }
I20260804 14:53:41.595064 5811 raft_consensus.cc:2866] T be8a881726c9437283de67de4c31559a P 3e653e9eb883427686d59e1134c442e1 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260804 14:53:41.595146 5811 raft_consensus.cc:2258] T be8a881726c9437283de67de4c31559a P 3e653e9eb883427686d59e1134c442e1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260804 14:53:41.595976 5811 ts_tablet_manager.cc:1905] T be8a881726c9437283de67de4c31559a P 3e653e9eb883427686d59e1134c442e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260804 14:53:41.598013 5811 ts_tablet_manager.cc:1918] T be8a881726c9437283de67de4c31559a P 3e653e9eb883427686d59e1134c442e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260804 14:53:41.598125 5811 log.cc:1196] T be8a881726c9437283de67de4c31559a P 3e653e9eb883427686d59e1134c442e1: Deleting WAL directory at /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/wals/be8a881726c9437283de67de4c31559a
I20260804 14:53:41.605252 5463 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5dcdea9fba5f4e908350c27a454d97ca"
dest_uuid: "3e653e9eb883427686d59e1134c442e1"
from {username='slave'} at 127.0.0.1:43062
I20260804 14:53:41.605480 5463 raft_consensus.cc:580] T 5dcdea9fba5f4e908350c27a454d97ca P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: Received request to step down
I20260804 14:53:41.605543 5463 raft_consensus.cc:3032] T 5dcdea9fba5f4e908350c27a454d97ca P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: Stepping down as leader of term 1
I20260804 14:53:41.605620 5463 raft_consensus.cc:729] T 5dcdea9fba5f4e908350c27a454d97ca P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3e653e9eb883427686d59e1134c442e1, State: Running, Role: LEADER
I20260804 14:53:41.605762 5463 consensus_queue.cc:260] T 5dcdea9fba5f4e908350c27a454d97ca P 3e653e9eb883427686d59e1134c442e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.605929 5463 raft_consensus.cc:3037] T 5dcdea9fba5f4e908350c27a454d97ca P 3e653e9eb883427686d59e1134c442e1 [term 1 FOLLOWER]: Advancing to term 2
I20260804 14:53:41.612563 5426 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dcdea9fba5f4e908350c27a454d97ca with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:43062
I20260804 14:53:41.612876 5811 tablet_replica.cc:330] T 5dcdea9fba5f4e908350c27a454d97ca P 3e653e9eb883427686d59e1134c442e1: stopping tablet replica
I20260804 14:53:41.613098 5811 raft_consensus.cc:2229] T 5dcdea9fba5f4e908350c27a454d97ca P 3e653e9eb883427686d59e1134c442e1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260804 14:53:41.613265 5811 pending_rounds.cc:62] T 5dcdea9fba5f4e908350c27a454d97ca P 3e653e9eb883427686d59e1134c442e1: Trying to abort 1 pending ops.
I20260804 14:53:41.613312 5811 pending_rounds.cc:68] T 5dcdea9fba5f4e908350c27a454d97ca P 3e653e9eb883427686d59e1134c442e1: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7314862987153276928 op_type: NO_OP noop_request { }
I20260804 14:53:41.613430 5811 raft_consensus.cc:2866] T 5dcdea9fba5f4e908350c27a454d97ca P 3e653e9eb883427686d59e1134c442e1 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260804 14:53:41.613509 5811 raft_consensus.cc:2258] T 5dcdea9fba5f4e908350c27a454d97ca P 3e653e9eb883427686d59e1134c442e1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260804 14:53:41.614277 5811 ts_tablet_manager.cc:1905] T 5dcdea9fba5f4e908350c27a454d97ca P 3e653e9eb883427686d59e1134c442e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260804 14:53:41.616690 5811 ts_tablet_manager.cc:1918] T 5dcdea9fba5f4e908350c27a454d97ca P 3e653e9eb883427686d59e1134c442e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260804 14:53:41.616814 5811 log.cc:1196] T 5dcdea9fba5f4e908350c27a454d97ca P 3e653e9eb883427686d59e1134c442e1: Deleting WAL directory at /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/wals/5dcdea9fba5f4e908350c27a454d97ca
I20260804 14:53:41.624526 5463 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "73efc26c877b46828c2b8ccadceaa91a"
dest_uuid: "3e653e9eb883427686d59e1134c442e1"
from {username='slave'} at 127.0.0.1:43062
I20260804 14:53:41.624782 5463 raft_consensus.cc:580] T 73efc26c877b46828c2b8ccadceaa91a P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: Received request to step down
I20260804 14:53:41.624853 5463 raft_consensus.cc:3032] T 73efc26c877b46828c2b8ccadceaa91a P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: Stepping down as leader of term 1
I20260804 14:53:41.624960 5463 raft_consensus.cc:729] T 73efc26c877b46828c2b8ccadceaa91a P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3e653e9eb883427686d59e1134c442e1, State: Running, Role: LEADER
I20260804 14:53:41.625123 5463 consensus_queue.cc:260] T 73efc26c877b46828c2b8ccadceaa91a P 3e653e9eb883427686d59e1134c442e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.625303 5463 raft_consensus.cc:3037] T 73efc26c877b46828c2b8ccadceaa91a P 3e653e9eb883427686d59e1134c442e1 [term 1 FOLLOWER]: Advancing to term 2
I20260804 14:53:41.632066 5426 tablet_service.cc:1514] Processing DeleteTablet for tablet 73efc26c877b46828c2b8ccadceaa91a with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:43062
I20260804 14:53:41.632365 5811 tablet_replica.cc:330] T 73efc26c877b46828c2b8ccadceaa91a P 3e653e9eb883427686d59e1134c442e1: stopping tablet replica
I20260804 14:53:41.632537 5811 raft_consensus.cc:2229] T 73efc26c877b46828c2b8ccadceaa91a P 3e653e9eb883427686d59e1134c442e1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260804 14:53:41.632702 5811 pending_rounds.cc:62] T 73efc26c877b46828c2b8ccadceaa91a P 3e653e9eb883427686d59e1134c442e1: Trying to abort 1 pending ops.
I20260804 14:53:41.632748 5811 pending_rounds.cc:68] T 73efc26c877b46828c2b8ccadceaa91a P 3e653e9eb883427686d59e1134c442e1: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7314862987213008896 op_type: NO_OP noop_request { }
I20260804 14:53:41.632859 5811 raft_consensus.cc:2866] T 73efc26c877b46828c2b8ccadceaa91a P 3e653e9eb883427686d59e1134c442e1 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260804 14:53:41.632931 5811 raft_consensus.cc:2258] T 73efc26c877b46828c2b8ccadceaa91a P 3e653e9eb883427686d59e1134c442e1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260804 14:53:41.633749 5811 ts_tablet_manager.cc:1905] T 73efc26c877b46828c2b8ccadceaa91a P 3e653e9eb883427686d59e1134c442e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260804 14:53:41.635934 5811 ts_tablet_manager.cc:1918] T 73efc26c877b46828c2b8ccadceaa91a P 3e653e9eb883427686d59e1134c442e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260804 14:53:41.636054 5811 log.cc:1196] T 73efc26c877b46828c2b8ccadceaa91a P 3e653e9eb883427686d59e1134c442e1: Deleting WAL directory at /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/wals/73efc26c877b46828c2b8ccadceaa91a
I20260804 14:53:41.643282 5463 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "57ee2bd239614464a99304a04ddc67bf"
dest_uuid: "3e653e9eb883427686d59e1134c442e1"
from {username='slave'} at 127.0.0.1:43062
I20260804 14:53:41.643580 5463 raft_consensus.cc:580] T 57ee2bd239614464a99304a04ddc67bf P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: Received request to step down
I20260804 14:53:41.643649 5463 raft_consensus.cc:3032] T 57ee2bd239614464a99304a04ddc67bf P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: Stepping down as leader of term 1
I20260804 14:53:41.643703 5463 raft_consensus.cc:729] T 57ee2bd239614464a99304a04ddc67bf P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3e653e9eb883427686d59e1134c442e1, State: Running, Role: LEADER
I20260804 14:53:41.643937 5463 consensus_queue.cc:260] T 57ee2bd239614464a99304a04ddc67bf P 3e653e9eb883427686d59e1134c442e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.644129 5463 raft_consensus.cc:3037] T 57ee2bd239614464a99304a04ddc67bf P 3e653e9eb883427686d59e1134c442e1 [term 1 FOLLOWER]: Advancing to term 2
I20260804 14:53:41.651751 5426 tablet_service.cc:1514] Processing DeleteTablet for tablet 57ee2bd239614464a99304a04ddc67bf with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:43062
I20260804 14:53:41.652083 5811 tablet_replica.cc:330] T 57ee2bd239614464a99304a04ddc67bf P 3e653e9eb883427686d59e1134c442e1: stopping tablet replica
I20260804 14:53:41.652280 5811 raft_consensus.cc:2229] T 57ee2bd239614464a99304a04ddc67bf P 3e653e9eb883427686d59e1134c442e1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260804 14:53:41.652449 5811 pending_rounds.cc:62] T 57ee2bd239614464a99304a04ddc67bf P 3e653e9eb883427686d59e1134c442e1: Trying to abort 1 pending ops.
I20260804 14:53:41.652498 5811 pending_rounds.cc:68] T 57ee2bd239614464a99304a04ddc67bf P 3e653e9eb883427686d59e1134c442e1: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7314862987268038656 op_type: NO_OP noop_request { }
I20260804 14:53:41.652627 5811 raft_consensus.cc:2866] T 57ee2bd239614464a99304a04ddc67bf P 3e653e9eb883427686d59e1134c442e1 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260804 14:53:41.652704 5811 raft_consensus.cc:2258] T 57ee2bd239614464a99304a04ddc67bf P 3e653e9eb883427686d59e1134c442e1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260804 14:53:41.653785 5811 ts_tablet_manager.cc:1905] T 57ee2bd239614464a99304a04ddc67bf P 3e653e9eb883427686d59e1134c442e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260804 14:53:41.656386 5811 ts_tablet_manager.cc:1918] T 57ee2bd239614464a99304a04ddc67bf P 3e653e9eb883427686d59e1134c442e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260804 14:53:41.656509 5811 log.cc:1196] T 57ee2bd239614464a99304a04ddc67bf P 3e653e9eb883427686d59e1134c442e1: Deleting WAL directory at /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/wals/57ee2bd239614464a99304a04ddc67bf
I20260804 14:53:41.663653 5463 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "335ce7f1cab04f2d8c23b89e30fca53d"
dest_uuid: "3e653e9eb883427686d59e1134c442e1"
from {username='slave'} at 127.0.0.1:43062
I20260804 14:53:41.663961 5463 raft_consensus.cc:580] T 335ce7f1cab04f2d8c23b89e30fca53d P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: Received request to step down
I20260804 14:53:41.664042 5463 raft_consensus.cc:3032] T 335ce7f1cab04f2d8c23b89e30fca53d P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: Stepping down as leader of term 1
I20260804 14:53:41.664089 5463 raft_consensus.cc:729] T 335ce7f1cab04f2d8c23b89e30fca53d P 3e653e9eb883427686d59e1134c442e1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3e653e9eb883427686d59e1134c442e1, State: Running, Role: LEADER
I20260804 14:53:41.664258 5463 consensus_queue.cc:260] T 335ce7f1cab04f2d8c23b89e30fca53d P 3e653e9eb883427686d59e1134c442e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e653e9eb883427686d59e1134c442e1" member_type: VOTER last_known_addr { host: "127.24.6.130" port: 38561 } } peers { permanent_uuid: "2fb91ff838c74c0490ce3abaca0e933b" member_type: VOTER last_known_addr { host: "127.24.6.129" port: 44579 } } peers { permanent_uuid: "61d01a203880481b9bed9ab4b6d4cd89" member_type: VOTER last_known_addr { host: "127.24.6.131" port: 44621 } }
I20260804 14:53:41.664453 5463 raft_consensus.cc:3037] T 335ce7f1cab04f2d8c23b89e30fca53d P 3e653e9eb883427686d59e1134c442e1 [term 1 FOLLOWER]: Advancing to term 2
I20260804 14:53:41.671329 5426 tablet_service.cc:1514] Processing DeleteTablet for tablet 335ce7f1cab04f2d8c23b89e30fca53d with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:43062
I20260804 14:53:41.671657 5811 tablet_replica.cc:330] T 335ce7f1cab04f2d8c23b89e30fca53d P 3e653e9eb883427686d59e1134c442e1: stopping tablet replica
I20260804 14:53:41.671833 5811 raft_consensus.cc:2229] T 335ce7f1cab04f2d8c23b89e30fca53d P 3e653e9eb883427686d59e1134c442e1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260804 14:53:41.672035 5811 pending_rounds.cc:62] T 335ce7f1cab04f2d8c23b89e30fca53d P 3e653e9eb883427686d59e1134c442e1: Trying to abort 1 pending ops.
I20260804 14:53:41.672093 5811 pending_rounds.cc:68] T 335ce7f1cab04f2d8c23b89e30fca53d P 3e653e9eb883427686d59e1134c442e1: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7314862987319816192 op_type: NO_OP noop_request { }
I20260804 14:53:41.672206 5811 raft_consensus.cc:2866] T 335ce7f1cab04f2d8c23b89e30fca53d P 3e653e9eb883427686d59e1134c442e1 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260804 14:53:41.672276 5811 raft_consensus.cc:2258] T 335ce7f1cab04f2d8c23b89e30fca53d P 3e653e9eb883427686d59e1134c442e1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260804 14:53:41.673118 5811 ts_tablet_manager.cc:1905] T 335ce7f1cab04f2d8c23b89e30fca53d P 3e653e9eb883427686d59e1134c442e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260804 14:53:41.675174 5811 ts_tablet_manager.cc:1918] T 335ce7f1cab04f2d8c23b89e30fca53d P 3e653e9eb883427686d59e1134c442e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260804 14:53:41.675326 5811 log.cc:1196] T 335ce7f1cab04f2d8c23b89e30fca53d P 3e653e9eb883427686d59e1134c442e1: Deleting WAL directory at /tmp/dist-test-taskp1wyor/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1785855048081067-24602-0/raft_consensus-itest-cluster/ts-1/wal/wals/335ce7f1cab04f2d8c23b89e30fca53d
I20260804 14:53:41.683503 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 5170
I20260804 14:53:41.701259 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 5349
I20260804 14:53:41.716722 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 5524
I20260804 14:53:41.730057 24602 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp1wyor/build/debug/bin/kudu with pid 5036
2026-08-04T14:53:41Z chronyd exiting
[ OK ] NumReplicas/RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric/1 (1525 ms)
[----------] 2 tests from NumReplicas/RaftConsensusNumLeadersMetricTest (2520 ms total)
[----------] Global test environment tear-down
[==========] 14 tests from 2 test suites ran. (173633 ms total)
[ PASSED ] 13 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] RaftConsensusElectionITest.TestNewLeaderCantResolvePeers
1 FAILED TEST
I20260804 14:53:41.770493 24602 logging.cc:424] LogThrottler /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 2 messages since previous log ~11 seconds ago