[==========] Running 14 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 12 tests from RaftConsensusElectionITest
[ RUN ] RaftConsensusElectionITest.TestNewLeaderCantResolvePeers
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260808 07:00:46.503636 16362 test_util.cc:268] Using random seed: -1372687697
I20260808 07:00:46.510140 16362 ts_itest-base.cc:112] Starting cluster with:
I20260808 07:00:46.510208 16362 ts_itest-base.cc:113] --------------
I20260808 07:00:46.510221 16362 ts_itest-base.cc:114] 3 tablet servers
I20260808 07:00:46.510238 16362 ts_itest-base.cc:115] 3 replicas per TS
I20260808 07:00:46.510249 16362 ts_itest-base.cc:116] --------------
2026-08-08T07:00:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:00:46Z Disabled control of system clock
I20260808 07:00:46.520556 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.250.190:40665
--webserver_interface=127.15.250.190
--webserver_port=0
--builtin_ntp_servers=127.15.250.148:45405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.250.190:40665
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20260808 07:00:46.602979 16374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:46.603209 16374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:46.603228 16374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:46.604483 16374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:00:46.604527 16374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:46.604542 16374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:00:46.604554 16374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:00:46.606004 16374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:45405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.15.250.190:40665
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.250.190:40665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.250.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.16374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:00:46.606244 16374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:46.606510 16374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:46.611784 16380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:46.611810 16383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:46.611851 16374 server_base.cc:1034] running on GCE node
W20260808 07:00:46.611821 16381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:46.612386 16374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:46.612707 16374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:46.613864 16374 hybrid_clock.cc:648] HybridClock initialized: now 1786172446613889 us; error 70 us; skew 500 ppm
I20260808 07:00:46.615276 16374 webserver.cc:460] Webserver started at http://127.15.250.190:33129/ using document root <none> and password file <none>
I20260808 07:00:46.615576 16374 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:46.615635 16374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:46.615789 16374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:00:46.616886 16374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/master-0/data/instance:
uuid: "7eb75227655a44e3b28d0bc307618744"
format_stamp: "Formatted at 2026-08-08 07:00:46 on dist-test-slave-0b58"
I20260808 07:00:46.617264 16374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/master-0/wal/instance:
uuid: "7eb75227655a44e3b28d0bc307618744"
format_stamp: "Formatted at 2026-08-08 07:00:46 on dist-test-slave-0b58"
I20260808 07:00:46.618710 16374 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 07:00:46.619524 16389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:46.619696 16374 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:00:46.619812 16374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/master-0/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/master-0/wal
uuid: "7eb75227655a44e3b28d0bc307618744"
format_stamp: "Formatted at 2026-08-08 07:00:46 on dist-test-slave-0b58"
I20260808 07:00:46.619889 16374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:46.644445 16374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:46.644829 16374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:46.645004 16374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:46.649039 16374 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.190:40665
I20260808 07:00:46.649076 16445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.190:40665 every 8 connection(s)
I20260808 07:00:46.649452 16374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/master-0/data/info.pb
I20260808 07:00:46.650122 16446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:00:46.652807 16446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eb75227655a44e3b28d0bc307618744: Bootstrap starting.
I20260808 07:00:46.653460 16446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7eb75227655a44e3b28d0bc307618744: Neither blocks nor log segments found. Creating new log.
I20260808 07:00:46.653766 16446 log.cc:824] T 00000000000000000000000000000000 P 7eb75227655a44e3b28d0bc307618744: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:46.654502 16446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eb75227655a44e3b28d0bc307618744: No bootstrap required, opened a new log
I20260808 07:00:46.655670 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 16374
I20260808 07:00:46.655761 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/master-0/wal/instance
I20260808 07:00:46.656198 16446 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7eb75227655a44e3b28d0bc307618744 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb75227655a44e3b28d0bc307618744" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 40665 } }
I20260808 07:00:46.656427 16446 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7eb75227655a44e3b28d0bc307618744 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:00:46.656483 16446 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7eb75227655a44e3b28d0bc307618744 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb75227655a44e3b28d0bc307618744, State: Initialized, Role: FOLLOWER
I20260808 07:00:46.656656 16446 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eb75227655a44e3b28d0bc307618744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb75227655a44e3b28d0bc307618744" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 40665 } }
I20260808 07:00:46.656774 16446 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7eb75227655a44e3b28d0bc307618744 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:00:46.656821 16446 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7eb75227655a44e3b28d0bc307618744 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:00:46.656872 16446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7eb75227655a44e3b28d0bc307618744 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:00:46.657416 16446 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7eb75227655a44e3b28d0bc307618744 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb75227655a44e3b28d0bc307618744" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 40665 } }
I20260808 07:00:46.657546 16446 leader_election.cc:302] T 00000000000000000000000000000000 P 7eb75227655a44e3b28d0bc307618744 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7eb75227655a44e3b28d0bc307618744; no voters:
I20260808 07:00:46.657706 16446 leader_election.cc:288] T 00000000000000000000000000000000 P 7eb75227655a44e3b28d0bc307618744 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:00:46.657814 16449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7eb75227655a44e3b28d0bc307618744 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:00:46.657999 16446 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7eb75227655a44e3b28d0bc307618744 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:00:46.658010 16449 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7eb75227655a44e3b28d0bc307618744 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb75227655a44e3b28d0bc307618744, State: Running, Role: LEADER
I20260808 07:00:46.658250 16449 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eb75227655a44e3b28d0bc307618744 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb75227655a44e3b28d0bc307618744" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 40665 } }
I20260808 07:00:46.658691 16451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb75227655a44e3b28d0bc307618744 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7eb75227655a44e3b28d0bc307618744" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb75227655a44e3b28d0bc307618744" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 40665 } } }
I20260808 07:00:46.658710 16450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb75227655a44e3b28d0bc307618744 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eb75227655a44e3b28d0bc307618744. Latest consensus state: current_term: 1 leader_uuid: "7eb75227655a44e3b28d0bc307618744" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb75227655a44e3b28d0bc307618744" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 40665 } } }
I20260808 07:00:46.658874 16451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb75227655a44e3b28d0bc307618744 [sys.catalog]: This master's current role is: LEADER
I20260808 07:00:46.658883 16450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb75227655a44e3b28d0bc307618744 [sys.catalog]: This master's current role is: LEADER
W20260808 07:00:46.660531 16467 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7eb75227655a44e3b28d0bc307618744: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 07:00:46.660605 16467 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 07:00:46.660666 16455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:00:46.660842 16455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:00:46.662497 16455 catalog_manager.cc:1347] Generated new cluster ID: a19d7a6fa4e14b36bb62486968bb4c35
I20260808 07:00:46.662562 16455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:00:46.670656 16455 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:00:46.671190 16455 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:00:46.677837 16455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7eb75227655a44e3b28d0bc307618744: Generated new TSK 0
I20260808 07:00:46.678045 16455 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:00:46.688212 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.129:0
--local_ip_for_outbound_sockets=127.15.250.129
--webserver_interface=127.15.250.129
--webserver_port=0
--tserver_master_addrs=127.15.250.190:40665
--builtin_ntp_servers=127.15.250.148:45405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--dns_resolver_cache_capacity_mb=0
--enable_leader_failure_detection=false
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=3
--leader_failure_max_missed_heartbeat_periods=10
--leader_failure_exp_backoff_max_delta_ms=5000 with env {}
W20260808 07:00:46.791004 16470 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 07:00:46.791231 16470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:46.791249 16470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:46.791268 16470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:46.792567 16470 flags.cc:425] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=5000
W20260808 07:00:46.792620 16470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:46.792647 16470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.129
I20260808 07:00:46.794276 16470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:45405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--enable_leader_failure_detection=false
--leader_failure_exp_backoff_max_delta_ms=5000
--leader_failure_max_missed_heartbeat_periods=10
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.250.129
--webserver_port=0
--tserver_master_addrs=127.15.250.190:40665
--never_fsync=true
--heap_profile_path=/tmp/kudu.16470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--dns_resolver_cache_capacity_mb=0
--local_ip_for_outbound_sockets=127.15.250.129
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:00:46.794574 16470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:46.794970 16470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:46.801148 16476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:46.801245 16475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:46.801390 16470 server_base.cc:1034] running on GCE node
W20260808 07:00:46.801126 16478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:46.801678 16470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:46.801950 16470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:46.803124 16470 hybrid_clock.cc:648] HybridClock initialized: now 1786172446803097 us; error 45 us; skew 500 ppm
I20260808 07:00:46.804626 16470 webserver.cc:460] Webserver started at http://127.15.250.129:46753/ using document root <none> and password file <none>
I20260808 07:00:46.804883 16470 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:46.804955 16470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:46.805115 16470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:00:46.806164 16470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-0/data/instance:
uuid: "793cd2aa95144e30aaa7a21ca56039c2"
format_stamp: "Formatted at 2026-08-08 07:00:46 on dist-test-slave-0b58"
I20260808 07:00:46.806581 16470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-0/wal/instance:
uuid: "793cd2aa95144e30aaa7a21ca56039c2"
format_stamp: "Formatted at 2026-08-08 07:00:46 on dist-test-slave-0b58"
I20260808 07:00:46.808044 16470 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 07:00:46.808960 16484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:46.809139 16470 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 07:00:46.809252 16470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-0/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-0/wal
uuid: "793cd2aa95144e30aaa7a21ca56039c2"
format_stamp: "Formatted at 2026-08-08 07:00:46 on dist-test-slave-0b58"
I20260808 07:00:46.809327 16470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:46.826838 16470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:46.827282 16470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:46.827462 16470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:46.827741 16470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:00:46.828111 16470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:00:46.828176 16470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:46.828222 16470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:00:46.828249 16470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:46.834204 16470 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.129:44453
I20260808 07:00:46.834285 16601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.129:44453 every 8 connection(s)
I20260808 07:00:46.834702 16470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-0/data/info.pb
I20260808 07:00:46.840905 16602 heartbeater.cc:347] Connected to a master server at 127.15.250.190:40665
I20260808 07:00:46.841017 16602 heartbeater.cc:464] Registering TS with master...
I20260808 07:00:46.841168 16602 heartbeater.cc:511] Master 127.15.250.190:40665 requested a full tablet report, sending...
I20260808 07:00:46.841727 16407 ts_manager.cc:194] Registered new tserver with Master: 793cd2aa95144e30aaa7a21ca56039c2 (127.15.250.129:44453)
I20260808 07:00:46.842375 16407 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.129:59159
I20260808 07:00:46.844856 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 16470
I20260808 07:00:46.844928 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-0/wal/instance
I20260808 07:00:46.846220 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.130:0
--local_ip_for_outbound_sockets=127.15.250.130
--webserver_interface=127.15.250.130
--webserver_port=0
--tserver_master_addrs=127.15.250.190:40665
--builtin_ntp_servers=127.15.250.148:45405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--dns_resolver_cache_capacity_mb=0
--enable_leader_failure_detection=false
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=3
--leader_failure_max_missed_heartbeat_periods=10
--leader_failure_exp_backoff_max_delta_ms=5000 with env {}
W20260808 07:00:46.931875 16605 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 07:00:46.932108 16605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:46.932125 16605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:46.932143 16605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:46.933342 16605 flags.cc:425] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=5000
W20260808 07:00:46.933390 16605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:46.933416 16605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.130
I20260808 07:00:46.934677 16605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:45405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--enable_leader_failure_detection=false
--leader_failure_exp_backoff_max_delta_ms=5000
--leader_failure_max_missed_heartbeat_periods=10
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.250.130
--webserver_port=0
--tserver_master_addrs=127.15.250.190:40665
--never_fsync=true
--heap_profile_path=/tmp/kudu.16605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--dns_resolver_cache_capacity_mb=0
--local_ip_for_outbound_sockets=127.15.250.130
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:00:46.934882 16605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:46.935134 16605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:46.940011 16610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:46.940022 16612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:46.940034 16614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:46.940284 16605 server_base.cc:1034] running on GCE node
I20260808 07:00:46.940500 16605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:46.940744 16605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:46.941931 16605 hybrid_clock.cc:648] HybridClock initialized: now 1786172446941925 us; error 38 us; skew 500 ppm
I20260808 07:00:46.943252 16605 webserver.cc:460] Webserver started at http://127.15.250.130:41473/ using document root <none> and password file <none>
I20260808 07:00:46.943526 16605 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:46.943598 16605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:46.943760 16605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:00:46.944885 16605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-1/data/instance:
uuid: "675705f02b8044a28cd6f540ecba998a"
format_stamp: "Formatted at 2026-08-08 07:00:46 on dist-test-slave-0b58"
I20260808 07:00:46.945264 16605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-1/wal/instance:
uuid: "675705f02b8044a28cd6f540ecba998a"
format_stamp: "Formatted at 2026-08-08 07:00:46 on dist-test-slave-0b58"
I20260808 07:00:46.946586 16605 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 07:00:46.947296 16620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:46.947564 16605 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 07:00:46.947674 16605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-1/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-1/wal
uuid: "675705f02b8044a28cd6f540ecba998a"
format_stamp: "Formatted at 2026-08-08 07:00:46 on dist-test-slave-0b58"
I20260808 07:00:46.947752 16605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:46.970691 16605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:46.971050 16605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:46.971220 16605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:46.971470 16605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:00:46.971846 16605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:00:46.971911 16605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:46.971956 16605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:00:46.972005 16605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:46.977744 16605 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.130:42869
I20260808 07:00:46.977833 16741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.130:42869 every 8 connection(s)
I20260808 07:00:46.978192 16605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-1/data/info.pb
I20260808 07:00:46.981858 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 16605
I20260808 07:00:46.981959 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-1/wal/instance
I20260808 07:00:46.983084 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.131:0
--local_ip_for_outbound_sockets=127.15.250.131
--webserver_interface=127.15.250.131
--webserver_port=0
--tserver_master_addrs=127.15.250.190:40665
--builtin_ntp_servers=127.15.250.148:45405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--dns_resolver_cache_capacity_mb=0
--enable_leader_failure_detection=false
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=3
--leader_failure_max_missed_heartbeat_periods=10
--leader_failure_exp_backoff_max_delta_ms=5000 with env {}
I20260808 07:00:46.984344 16742 heartbeater.cc:347] Connected to a master server at 127.15.250.190:40665
I20260808 07:00:46.984459 16742 heartbeater.cc:464] Registering TS with master...
I20260808 07:00:46.984669 16742 heartbeater.cc:511] Master 127.15.250.190:40665 requested a full tablet report, sending...
I20260808 07:00:46.985056 16407 ts_manager.cc:194] Registered new tserver with Master: 675705f02b8044a28cd6f540ecba998a (127.15.250.130:42869)
I20260808 07:00:46.985435 16407 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.130:47461
W20260808 07:00:47.066699 16745 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 07:00:47.066962 16745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:47.066991 16745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:47.067032 16745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:47.068311 16745 flags.cc:425] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=5000
W20260808 07:00:47.068374 16745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:47.068418 16745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.131
I20260808 07:00:47.069761 16745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:45405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--enable_leader_failure_detection=false
--leader_failure_exp_backoff_max_delta_ms=5000
--leader_failure_max_missed_heartbeat_periods=10
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.250.131
--webserver_port=0
--tserver_master_addrs=127.15.250.190:40665
--never_fsync=true
--heap_profile_path=/tmp/kudu.16745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--dns_resolver_cache_capacity_mb=0
--local_ip_for_outbound_sockets=127.15.250.131
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:00:47.070026 16745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:47.070328 16745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:47.075251 16753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:47.075354 16751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:47.075357 16750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:47.075872 16745 server_base.cc:1034] running on GCE node
I20260808 07:00:47.076061 16745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:47.076352 16745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:47.077504 16745 hybrid_clock.cc:648] HybridClock initialized: now 1786172447077457 us; error 64 us; skew 500 ppm
I20260808 07:00:47.078807 16745 webserver.cc:460] Webserver started at http://127.15.250.131:44375/ using document root <none> and password file <none>
I20260808 07:00:47.079049 16745 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:47.079095 16745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:47.079238 16745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:00:47.080214 16745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-2/data/instance:
uuid: "5593115523c94df5bb97f3c3a53dc5b5"
format_stamp: "Formatted at 2026-08-08 07:00:47 on dist-test-slave-0b58"
I20260808 07:00:47.080653 16745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-2/wal/instance:
uuid: "5593115523c94df5bb97f3c3a53dc5b5"
format_stamp: "Formatted at 2026-08-08 07:00:47 on dist-test-slave-0b58"
I20260808 07:00:47.082016 16745 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:00:47.082808 16759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:47.083009 16745 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:00:47.083109 16745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-2/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-2/wal
uuid: "5593115523c94df5bb97f3c3a53dc5b5"
format_stamp: "Formatted at 2026-08-08 07:00:47 on dist-test-slave-0b58"
I20260808 07:00:47.083189 16745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:47.097659 16745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:47.097993 16745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:47.098146 16745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:47.098387 16745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:00:47.098764 16745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:00:47.098816 16745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:47.098852 16745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:00:47.098879 16745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:47.104862 16745 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.131:34957
I20260808 07:00:47.104911 16877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.131:34957 every 8 connection(s)
I20260808 07:00:47.105255 16745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-2/data/info.pb
I20260808 07:00:47.108824 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 16745
I20260808 07:00:47.108924 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-2/wal/instance
I20260808 07:00:47.111496 16878 heartbeater.cc:347] Connected to a master server at 127.15.250.190:40665
I20260808 07:00:47.111598 16878 heartbeater.cc:464] Registering TS with master...
I20260808 07:00:47.111792 16878 heartbeater.cc:511] Master 127.15.250.190:40665 requested a full tablet report, sending...
I20260808 07:00:47.112169 16407 ts_manager.cc:194] Registered new tserver with Master: 5593115523c94df5bb97f3c3a53dc5b5 (127.15.250.131:34957)
I20260808 07:00:47.112653 16407 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.131:34755
I20260808 07:00:47.121385 16362 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 07:00:47.122824 16362 test_util.cc:268] Using random seed: -1372068475
I20260808 07:00:47.131294 16407 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53000:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 07:00:47.131655 16407 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 07:00:47.138653 16670 tablet_service.cc:1467] Processing CreateTablet for tablet 9cdab77050924e1ca1cec06b14f932a6 (DEFAULT_TABLE table=test-workload [id=12ed5a81caa84c538b2a80aaeb5d1b02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:00:47.138733 16535 tablet_service.cc:1467] Processing CreateTablet for tablet 9cdab77050924e1ca1cec06b14f932a6 (DEFAULT_TABLE table=test-workload [id=12ed5a81caa84c538b2a80aaeb5d1b02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:00:47.139091 16670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cdab77050924e1ca1cec06b14f932a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:00:47.139091 16535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cdab77050924e1ca1cec06b14f932a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:00:47.139017 16808 tablet_service.cc:1467] Processing CreateTablet for tablet 9cdab77050924e1ca1cec06b14f932a6 (DEFAULT_TABLE table=test-workload [id=12ed5a81caa84c538b2a80aaeb5d1b02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:00:47.139292 16808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cdab77050924e1ca1cec06b14f932a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:00:47.141808 16892 tablet_bootstrap.cc:492] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5: Bootstrap starting.
I20260808 07:00:47.142208 16894 tablet_bootstrap.cc:492] T 9cdab77050924e1ca1cec06b14f932a6 P 675705f02b8044a28cd6f540ecba998a: Bootstrap starting.
I20260808 07:00:47.142494 16892 tablet_bootstrap.cc:654] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5: Neither blocks nor log segments found. Creating new log.
I20260808 07:00:47.142843 16892 log.cc:824] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:47.142876 16894 tablet_bootstrap.cc:654] T 9cdab77050924e1ca1cec06b14f932a6 P 675705f02b8044a28cd6f540ecba998a: Neither blocks nor log segments found. Creating new log.
I20260808 07:00:47.143187 16894 log.cc:824] T 9cdab77050924e1ca1cec06b14f932a6 P 675705f02b8044a28cd6f540ecba998a: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:47.143306 16893 tablet_bootstrap.cc:492] T 9cdab77050924e1ca1cec06b14f932a6 P 793cd2aa95144e30aaa7a21ca56039c2: Bootstrap starting.
I20260808 07:00:47.143733 16892 tablet_bootstrap.cc:492] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5: No bootstrap required, opened a new log
I20260808 07:00:47.143832 16892 ts_tablet_manager.cc:1397] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:00:47.143997 16893 tablet_bootstrap.cc:654] T 9cdab77050924e1ca1cec06b14f932a6 P 793cd2aa95144e30aaa7a21ca56039c2: Neither blocks nor log segments found. Creating new log.
I20260808 07:00:47.144338 16893 log.cc:824] T 9cdab77050924e1ca1cec06b14f932a6 P 793cd2aa95144e30aaa7a21ca56039c2: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:47.144779 16894 tablet_bootstrap.cc:492] T 9cdab77050924e1ca1cec06b14f932a6 P 675705f02b8044a28cd6f540ecba998a: No bootstrap required, opened a new log
I20260808 07:00:47.144860 16894 ts_tablet_manager.cc:1397] T 9cdab77050924e1ca1cec06b14f932a6 P 675705f02b8044a28cd6f540ecba998a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260808 07:00:47.145117 16893 tablet_bootstrap.cc:492] T 9cdab77050924e1ca1cec06b14f932a6 P 793cd2aa95144e30aaa7a21ca56039c2: No bootstrap required, opened a new log
I20260808 07:00:47.145195 16893 ts_tablet_manager.cc:1397] T 9cdab77050924e1ca1cec06b14f932a6 P 793cd2aa95144e30aaa7a21ca56039c2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 07:00:47.146729 16892 raft_consensus.cc:348] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793cd2aa95144e30aaa7a21ca56039c2" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 44453 } } peers { permanent_uuid: "5593115523c94df5bb97f3c3a53dc5b5" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34957 } } peers { permanent_uuid: "675705f02b8044a28cd6f540ecba998a" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 42869 } }
I20260808 07:00:47.146929 16892 raft_consensus.cc:729] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5593115523c94df5bb97f3c3a53dc5b5, State: Initialized, Role: FOLLOWER
I20260808 07:00:47.147022 16892 consensus_queue.cc:260] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793cd2aa95144e30aaa7a21ca56039c2" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 44453 } } peers { permanent_uuid: "5593115523c94df5bb97f3c3a53dc5b5" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34957 } } peers { permanent_uuid: "675705f02b8044a28cd6f540ecba998a" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 42869 } }
I20260808 07:00:47.147030 16894 raft_consensus.cc:348] T 9cdab77050924e1ca1cec06b14f932a6 P 675705f02b8044a28cd6f540ecba998a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793cd2aa95144e30aaa7a21ca56039c2" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 44453 } } peers { permanent_uuid: "5593115523c94df5bb97f3c3a53dc5b5" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34957 } } peers { permanent_uuid: "675705f02b8044a28cd6f540ecba998a" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 42869 } }
I20260808 07:00:47.147228 16894 raft_consensus.cc:729] T 9cdab77050924e1ca1cec06b14f932a6 P 675705f02b8044a28cd6f540ecba998a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 675705f02b8044a28cd6f540ecba998a, State: Initialized, Role: FOLLOWER
I20260808 07:00:47.147300 16892 ts_tablet_manager.cc:1428] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260808 07:00:47.147312 16894 consensus_queue.cc:260] T 9cdab77050924e1ca1cec06b14f932a6 P 675705f02b8044a28cd6f540ecba998a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793cd2aa95144e30aaa7a21ca56039c2" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 44453 } } peers { permanent_uuid: "5593115523c94df5bb97f3c3a53dc5b5" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34957 } } peers { permanent_uuid: "675705f02b8044a28cd6f540ecba998a" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 42869 } }
I20260808 07:00:47.147439 16878 heartbeater.cc:503] Master 127.15.250.190:40665 was elected leader, sending a full tablet report...
I20260808 07:00:47.147579 16894 ts_tablet_manager.cc:1428] T 9cdab77050924e1ca1cec06b14f932a6 P 675705f02b8044a28cd6f540ecba998a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 07:00:47.147622 16742 heartbeater.cc:503] Master 127.15.250.190:40665 was elected leader, sending a full tablet report...
I20260808 07:00:47.148183 16893 raft_consensus.cc:348] T 9cdab77050924e1ca1cec06b14f932a6 P 793cd2aa95144e30aaa7a21ca56039c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793cd2aa95144e30aaa7a21ca56039c2" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 44453 } } peers { permanent_uuid: "5593115523c94df5bb97f3c3a53dc5b5" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34957 } } peers { permanent_uuid: "675705f02b8044a28cd6f540ecba998a" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 42869 } }
I20260808 07:00:47.148392 16893 raft_consensus.cc:729] T 9cdab77050924e1ca1cec06b14f932a6 P 793cd2aa95144e30aaa7a21ca56039c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793cd2aa95144e30aaa7a21ca56039c2, State: Initialized, Role: FOLLOWER
I20260808 07:00:47.148473 16893 consensus_queue.cc:260] T 9cdab77050924e1ca1cec06b14f932a6 P 793cd2aa95144e30aaa7a21ca56039c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793cd2aa95144e30aaa7a21ca56039c2" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 44453 } } peers { permanent_uuid: "5593115523c94df5bb97f3c3a53dc5b5" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34957 } } peers { permanent_uuid: "675705f02b8044a28cd6f540ecba998a" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 42869 } }
I20260808 07:00:47.148725 16893 ts_tablet_manager.cc:1428] T 9cdab77050924e1ca1cec06b14f932a6 P 793cd2aa95144e30aaa7a21ca56039c2: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260808 07:00:47.148871 16602 heartbeater.cc:503] Master 127.15.250.190:40665 was elected leader, sending a full tablet report...
W20260808 07:00:47.167160 16734 generic_service.cc:139] {username='slave'} at 127.0.0.1:33052 forcing change of non-runtime-safe flag fail_dns_resolution_hostports
I20260808 07:00:47.167253 16734 generic_service.cc:175] {username='slave'} at 127.0.0.1:33052 changed flags via RPC: fail_dns_resolution_hostports from '' to '127.15.250.130:42869'
W20260808 07:00:47.167902 16734 generic_service.cc:139] {username='slave'} at 127.0.0.1:33052 forcing change of non-runtime-safe flag fail_dns_resolution
I20260808 07:00:47.167953 16734 generic_service.cc:175] {username='slave'} at 127.0.0.1:33052 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
W20260808 07:00:47.171200 16872 generic_service.cc:139] {username='slave'} at 127.0.0.1:58782 forcing change of non-runtime-safe flag fail_dns_resolution_hostports
I20260808 07:00:47.171279 16872 generic_service.cc:175] {username='slave'} at 127.0.0.1:58782 changed flags via RPC: fail_dns_resolution_hostports from '' to '127.15.250.130:42869'
W20260808 07:00:47.171653 16872 generic_service.cc:139] {username='slave'} at 127.0.0.1:58782 forcing change of non-runtime-safe flag fail_dns_resolution
I20260808 07:00:47.171694 16872 generic_service.cc:175] {username='slave'} at 127.0.0.1:58782 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
W20260808 07:00:47.174908 16596 generic_service.cc:139] {username='slave'} at 127.0.0.1:55064 forcing change of non-runtime-safe flag fail_dns_resolution_hostports
I20260808 07:00:47.174988 16596 generic_service.cc:175] {username='slave'} at 127.0.0.1:55064 changed flags via RPC: fail_dns_resolution_hostports from '' to '127.15.250.130:42869'
W20260808 07:00:47.175498 16596 generic_service.cc:139] {username='slave'} at 127.0.0.1:55064 forcing change of non-runtime-safe flag fail_dns_resolution
I20260808 07:00:47.175546 16596 generic_service.cc:175] {username='slave'} at 127.0.0.1:55064 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
I20260808 07:00:47.176175 16829 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9cdab77050924e1ca1cec06b14f932a6"
dest_uuid: "5593115523c94df5bb97f3c3a53dc5b5"
from {username='slave'} at 127.0.0.1:58774
I20260808 07:00:47.176321 16829 raft_consensus.cc:482] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 07:00:47.176401 16829 raft_consensus.cc:3037] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:00:47.177115 16829 raft_consensus.cc:504] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793cd2aa95144e30aaa7a21ca56039c2" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 44453 } } peers { permanent_uuid: "5593115523c94df5bb97f3c3a53dc5b5" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34957 } } peers { permanent_uuid: "675705f02b8044a28cd6f540ecba998a" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 42869 } }
I20260808 07:00:47.177630 16829 leader_election.cc:288] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 [CANDIDATE]: Term 1 election: Requested vote from peers 793cd2aa95144e30aaa7a21ca56039c2 (127.15.250.129:44453), 675705f02b8044a28cd6f540ecba998a (127.15.250.130:42869)
W20260808 07:00:47.177732 16903 leader_election.cc:334] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 675705f02b8044a28cd6f540ecba998a (127.15.250.130:42869): Network error: failed to refresh physical address: injected DNS resolution failure: 127.15.250.130:42869
I20260808 07:00:47.180935 16555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cdab77050924e1ca1cec06b14f932a6" candidate_uuid: "5593115523c94df5bb97f3c3a53dc5b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "793cd2aa95144e30aaa7a21ca56039c2"
I20260808 07:00:47.181053 16555 raft_consensus.cc:3037] T 9cdab77050924e1ca1cec06b14f932a6 P 793cd2aa95144e30aaa7a21ca56039c2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:00:47.181850 16555 raft_consensus.cc:2454] T 9cdab77050924e1ca1cec06b14f932a6 P 793cd2aa95144e30aaa7a21ca56039c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5593115523c94df5bb97f3c3a53dc5b5 in term 1.
I20260808 07:00:47.182227 16763 leader_election.cc:302] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5593115523c94df5bb97f3c3a53dc5b5, 793cd2aa95144e30aaa7a21ca56039c2; no voters: 675705f02b8044a28cd6f540ecba998a
I20260808 07:00:47.182454 16898 raft_consensus.cc:2781] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:00:47.182698 16898 raft_consensus.cc:686] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 [term 1 LEADER]: Becoming Leader. State: Replica: 5593115523c94df5bb97f3c3a53dc5b5, State: Running, Role: LEADER
I20260808 07:00:47.182847 16898 consensus_queue.cc:237] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793cd2aa95144e30aaa7a21ca56039c2" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 44453 } } peers { permanent_uuid: "5593115523c94df5bb97f3c3a53dc5b5" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34957 } } peers { permanent_uuid: "675705f02b8044a28cd6f540ecba998a" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 42869 } }
I20260808 07:00:47.183873 16406 catalog_manager.cc:5495] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5593115523c94df5bb97f3c3a53dc5b5 (127.15.250.131). New cstate: current_term: 1 leader_uuid: "5593115523c94df5bb97f3c3a53dc5b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793cd2aa95144e30aaa7a21ca56039c2" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 44453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5593115523c94df5bb97f3c3a53dc5b5" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675705f02b8044a28cd6f540ecba998a" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 42869 } health_report { overall_health: UNKNOWN } } }
W20260808 07:00:47.228811 16743 tablet.cc:2369] T 9cdab77050924e1ca1cec06b14f932a6 P 675705f02b8044a28cd6f540ecba998a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 07:00:47.292196 16903 consensus_peers.cc:487] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 -> Peer 675705f02b8044a28cd6f540ecba998a (127.15.250.130:42869): Couldn't send request to peer 675705f02b8044a28cd6f540ecba998a. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.15.250.130:42869. This is attempt 1: this message will repeat every 5th retry.
I20260808 07:00:47.293898 16898 consensus_queue.cc:1035] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793cd2aa95144e30aaa7a21ca56039c2" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 44453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 07:00:47.800189 16903 consensus_peers.cc:487] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 -> Peer 675705f02b8044a28cd6f540ecba998a (127.15.250.130:42869): Couldn't send request to peer 675705f02b8044a28cd6f540ecba998a. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.15.250.130:42869. This is attempt 6: this message will repeat every 5th retry.
W20260808 07:00:48.252957 16903 consensus_peers.cc:487] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 -> Peer 675705f02b8044a28cd6f540ecba998a (127.15.250.130:42869): Couldn't send request to peer 675705f02b8044a28cd6f540ecba998a. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.15.250.130:42869. This is attempt 11: this message will repeat every 5th retry.
W20260808 07:00:48.738772 16903 consensus_peers.cc:487] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 -> Peer 675705f02b8044a28cd6f540ecba998a (127.15.250.130:42869): Couldn't send request to peer 675705f02b8044a28cd6f540ecba998a. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.15.250.130:42869. This is attempt 16: this message will repeat every 5th retry.
W20260808 07:00:49.217489 16903 consensus_peers.cc:487] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 -> Peer 675705f02b8044a28cd6f540ecba998a (127.15.250.130:42869): Couldn't send request to peer 675705f02b8044a28cd6f540ecba998a. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.15.250.130:42869. This is attempt 21: this message will repeat every 5th retry.
W20260808 07:00:49.676285 16903 consensus_peers.cc:487] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 -> Peer 675705f02b8044a28cd6f540ecba998a (127.15.250.130:42869): Couldn't send request to peer 675705f02b8044a28cd6f540ecba998a. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.15.250.130:42869. This is attempt 26: this message will repeat every 5th retry.
W20260808 07:00:50.175835 16903 consensus_peers.cc:487] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 -> Peer 675705f02b8044a28cd6f540ecba998a (127.15.250.130:42869): Couldn't send request to peer 675705f02b8044a28cd6f540ecba998a. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.15.250.130:42869. This is attempt 31: this message will repeat every 5th retry.
I20260808 07:00:50.299037 16898 consensus_queue.cc:579] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 [LEADER]: Leader has been unable to successfully communicate with peer 675705f02b8044a28cd6f540ecba998a for more than 3 seconds (3.116s)
W20260808 07:00:50.299722 16405 catalog_manager.cc:5173] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9cdab77050924e1ca1cec06b14f932a6 with cas_config_opid_index -1: no extra replica candidate found for tablet 9cdab77050924e1ca1cec06b14f932a6 (table test-workload [id=12ed5a81caa84c538b2a80aaeb5d1b02]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
W20260808 07:00:50.693796 16903 consensus_peers.cc:487] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 -> Peer 675705f02b8044a28cd6f540ecba998a (127.15.250.130:42869): Couldn't send request to peer 675705f02b8044a28cd6f540ecba998a. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.15.250.130:42869. This is attempt 36: this message will repeat every 5th retry.
W20260808 07:00:51.155082 16903 consensus_peers.cc:487] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 -> Peer 675705f02b8044a28cd6f540ecba998a (127.15.250.130:42869): Couldn't send request to peer 675705f02b8044a28cd6f540ecba998a. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.15.250.130:42869. This is attempt 41: this message will repeat every 5th retry.
W20260808 07:00:51.236334 16734 generic_service.cc:139] {username='slave'} at 127.0.0.1:33052 forcing change of non-runtime-safe flag fail_dns_resolution
I20260808 07:00:51.236419 16734 generic_service.cc:175] {username='slave'} at 127.0.0.1:33052 changed flags via RPC: fail_dns_resolution from 'true' to 'false'
W20260808 07:00:51.236901 16872 generic_service.cc:139] {username='slave'} at 127.0.0.1:58782 forcing change of non-runtime-safe flag fail_dns_resolution
I20260808 07:00:51.236959 16872 generic_service.cc:175] {username='slave'} at 127.0.0.1:58782 changed flags via RPC: fail_dns_resolution from 'true' to 'false'
W20260808 07:00:51.237399 16596 generic_service.cc:139] {username='slave'} at 127.0.0.1:55064 forcing change of non-runtime-safe flag fail_dns_resolution
I20260808 07:00:51.237453 16596 generic_service.cc:175] {username='slave'} at 127.0.0.1:55064 changed flags via RPC: fail_dns_resolution from 'true' to 'false'
I20260808 07:00:51.251552 16691 raft_consensus.cc:3037] T 9cdab77050924e1ca1cec06b14f932a6 P 675705f02b8044a28cd6f540ecba998a [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:00:51.378798 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.132:0
--local_ip_for_outbound_sockets=127.15.250.132
--webserver_interface=127.15.250.132
--webserver_port=0
--tserver_master_addrs=127.15.250.190:40665
--builtin_ntp_servers=127.15.250.148:45405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--dns_resolver_cache_capacity_mb=0
--enable_leader_failure_detection=false
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=3
--leader_failure_max_missed_heartbeat_periods=10
--leader_failure_exp_backoff_max_delta_ms=5000 with env {}
W20260808 07:00:51.463443 16919 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 07:00:51.463689 16919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:51.463718 16919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:51.463755 16919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:51.464962 16919 flags.cc:425] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=5000
W20260808 07:00:51.465036 16919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:51.465080 16919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.132
I20260808 07:00:51.466495 16919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:45405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--enable_leader_failure_detection=false
--leader_failure_exp_backoff_max_delta_ms=5000
--leader_failure_max_missed_heartbeat_periods=10
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.15.250.132
--webserver_port=0
--tserver_master_addrs=127.15.250.190:40665
--never_fsync=true
--heap_profile_path=/tmp/kudu.16919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--dns_resolver_cache_capacity_mb=0
--local_ip_for_outbound_sockets=127.15.250.132
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:00:51.466775 16919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:51.467093 16919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:51.472030 16927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:51.472049 16925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:51.472244 16919 server_base.cc:1034] running on GCE node
W20260808 07:00:51.472088 16924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:51.472568 16919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:51.472777 16919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:51.473915 16919 hybrid_clock.cc:648] HybridClock initialized: now 1786172451473896 us; error 35 us; skew 500 ppm
I20260808 07:00:51.479314 16919 webserver.cc:460] Webserver started at http://127.15.250.132:32939/ using document root <none> and password file <none>
I20260808 07:00:51.479555 16919 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:51.479601 16919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:51.479696 16919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:00:51.480736 16919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-3/data/instance:
uuid: "b51ae32d29b94b97a6037b9f890ba291"
format_stamp: "Formatted at 2026-08-08 07:00:51 on dist-test-slave-0b58"
I20260808 07:00:51.481041 16919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-3/wal/instance:
uuid: "b51ae32d29b94b97a6037b9f890ba291"
format_stamp: "Formatted at 2026-08-08 07:00:51 on dist-test-slave-0b58"
I20260808 07:00:51.482429 16919 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260808 07:00:51.483147 16934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:51.483389 16919 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:00:51.483482 16919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-3/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-3/wal
uuid: "b51ae32d29b94b97a6037b9f890ba291"
format_stamp: "Formatted at 2026-08-08 07:00:51 on dist-test-slave-0b58"
I20260808 07:00:51.483572 16919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:51.502031 16919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:51.502324 16919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:51.502429 16919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:51.502626 16919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:00:51.503006 16919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:00:51.503042 16919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:51.503067 16919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:00:51.503114 16919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:51.509291 16919 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.132:38085
I20260808 07:00:51.509382 17052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.132:38085 every 8 connection(s)
I20260808 07:00:51.509774 16919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-3/data/info.pb
I20260808 07:00:51.514616 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 16919
I20260808 07:00:51.514760 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0/raft_consensus_election-itest-cluster/ts-3/wal/instance
I20260808 07:00:51.516165 17053 heartbeater.cc:347] Connected to a master server at 127.15.250.190:40665
I20260808 07:00:51.516300 17053 heartbeater.cc:464] Registering TS with master...
I20260808 07:00:51.516486 17053 heartbeater.cc:511] Master 127.15.250.190:40665 requested a full tablet report, sending...
I20260808 07:00:51.516876 16405 ts_manager.cc:194] Registered new tserver with Master: b51ae32d29b94b97a6037b9f890ba291 (127.15.250.132:38085)
I20260808 07:00:51.517303 16405 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.132:57295
I20260808 07:00:51.517802 16829 consensus_queue.cc:237] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793cd2aa95144e30aaa7a21ca56039c2" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 44453 } } peers { permanent_uuid: "5593115523c94df5bb97f3c3a53dc5b5" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34957 } } peers { permanent_uuid: "675705f02b8044a28cd6f540ecba998a" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 42869 } } peers { permanent_uuid: "b51ae32d29b94b97a6037b9f890ba291" member_type: NON_VOTER last_known_addr { host: "127.15.250.132" port: 38085 } attrs { promote: true } }
I20260808 07:00:51.518546 16555 raft_consensus.cc:1261] T 9cdab77050924e1ca1cec06b14f932a6 P 793cd2aa95144e30aaa7a21ca56039c2 [term 1 FOLLOWER]: Refusing update from remote peer 5593115523c94df5bb97f3c3a53dc5b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 07:00:51.518800 16691 raft_consensus.cc:1261] T 9cdab77050924e1ca1cec06b14f932a6 P 675705f02b8044a28cd6f540ecba998a [term 1 FOLLOWER]: Refusing update from remote peer 5593115523c94df5bb97f3c3a53dc5b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 07:00:51.518780 16918 consensus_queue.cc:1035] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793cd2aa95144e30aaa7a21ca56039c2" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 44453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 07:00:51.519058 16918 consensus_queue.cc:1035] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675705f02b8044a28cd6f540ecba998a" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 42869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 07:00:51.519726 16917 raft_consensus.cc:2932] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b51ae32d29b94b97a6037b9f890ba291 (127.15.250.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793cd2aa95144e30aaa7a21ca56039c2" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 44453 } } peers { permanent_uuid: "5593115523c94df5bb97f3c3a53dc5b5" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34957 } } peers { permanent_uuid: "675705f02b8044a28cd6f540ecba998a" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 42869 } } peers { permanent_uuid: "b51ae32d29b94b97a6037b9f890ba291" member_type: NON_VOTER last_known_addr { host: "127.15.250.132" port: 38085 } attrs { promote: true } } }
I20260808 07:00:51.519824 16555 raft_consensus.cc:2932] T 9cdab77050924e1ca1cec06b14f932a6 P 793cd2aa95144e30aaa7a21ca56039c2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b51ae32d29b94b97a6037b9f890ba291 (127.15.250.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793cd2aa95144e30aaa7a21ca56039c2" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 44453 } } peers { permanent_uuid: "5593115523c94df5bb97f3c3a53dc5b5" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34957 } } peers { permanent_uuid: "675705f02b8044a28cd6f540ecba998a" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 42869 } } peers { permanent_uuid: "b51ae32d29b94b97a6037b9f890ba291" member_type: NON_VOTER last_known_addr { host: "127.15.250.132" port: 38085 } attrs { promote: true } } }
I20260808 07:00:51.521184 16405 catalog_manager.cc:5495] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 reported cstate change: config changed from index -1 to 2, NON_VOTER b51ae32d29b94b97a6037b9f890ba291 (127.15.250.132) added. New cstate: current_term: 1 leader_uuid: "5593115523c94df5bb97f3c3a53dc5b5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793cd2aa95144e30aaa7a21ca56039c2" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 44453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5593115523c94df5bb97f3c3a53dc5b5" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675705f02b8044a28cd6f540ecba998a" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 42869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b51ae32d29b94b97a6037b9f890ba291" member_type: NON_VOTER last_known_addr { host: "127.15.250.132" port: 38085 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 07:00:51.521627 16392 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9cdab77050924e1ca1cec06b14f932a6 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 7)
I20260808 07:00:51.521816 16691 raft_consensus.cc:2932] T 9cdab77050924e1ca1cec06b14f932a6 P 675705f02b8044a28cd6f540ecba998a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b51ae32d29b94b97a6037b9f890ba291 (127.15.250.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793cd2aa95144e30aaa7a21ca56039c2" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 44453 } } peers { permanent_uuid: "5593115523c94df5bb97f3c3a53dc5b5" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34957 } } peers { permanent_uuid: "675705f02b8044a28cd6f540ecba998a" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 42869 } } peers { permanent_uuid: "b51ae32d29b94b97a6037b9f890ba291" member_type: NON_VOTER last_known_addr { host: "127.15.250.132" port: 38085 } attrs { promote: true } } }
W20260808 07:00:51.523032 16761 consensus_peers.cc:487] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 -> Peer b51ae32d29b94b97a6037b9f890ba291 (127.15.250.132:38085): Couldn't send request to peer b51ae32d29b94b97a6037b9f890ba291. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9cdab77050924e1ca1cec06b14f932a6. This is attempt 1: this message will repeat every 5th retry.
I20260808 07:00:51.523936 16829 consensus_queue.cc:237] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793cd2aa95144e30aaa7a21ca56039c2" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 44453 } } peers { permanent_uuid: "5593115523c94df5bb97f3c3a53dc5b5" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34957 } } peers { permanent_uuid: "675705f02b8044a28cd6f540ecba998a" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 42869 } }
I20260808 07:00:51.524446 16555 raft_consensus.cc:1261] T 9cdab77050924e1ca1cec06b14f932a6 P 793cd2aa95144e30aaa7a21ca56039c2 [term 1 FOLLOWER]: Refusing update from remote peer 5593115523c94df5bb97f3c3a53dc5b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 07:00:51.524446 16691 raft_consensus.cc:1261] T 9cdab77050924e1ca1cec06b14f932a6 P 675705f02b8044a28cd6f540ecba998a [term 1 FOLLOWER]: Refusing update from remote peer 5593115523c94df5bb97f3c3a53dc5b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 07:00:51.524641 16918 consensus_queue.cc:1035] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793cd2aa95144e30aaa7a21ca56039c2" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 44453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 07:00:51.524727 16910 consensus_queue.cc:1035] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675705f02b8044a28cd6f540ecba998a" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 42869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 07:00:51.525282 16918 raft_consensus.cc:2932] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER b51ae32d29b94b97a6037b9f890ba291 (127.15.250.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793cd2aa95144e30aaa7a21ca56039c2" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 44453 } } peers { permanent_uuid: "5593115523c94df5bb97f3c3a53dc5b5" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34957 } } peers { permanent_uuid: "675705f02b8044a28cd6f540ecba998a" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 42869 } } }
I20260808 07:00:51.525434 16555 raft_consensus.cc:2932] T 9cdab77050924e1ca1cec06b14f932a6 P 793cd2aa95144e30aaa7a21ca56039c2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER b51ae32d29b94b97a6037b9f890ba291 (127.15.250.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793cd2aa95144e30aaa7a21ca56039c2" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 44453 } } peers { permanent_uuid: "5593115523c94df5bb97f3c3a53dc5b5" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34957 } } peers { permanent_uuid: "675705f02b8044a28cd6f540ecba998a" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 42869 } } }
I20260808 07:00:51.525437 16691 raft_consensus.cc:2932] T 9cdab77050924e1ca1cec06b14f932a6 P 675705f02b8044a28cd6f540ecba998a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER b51ae32d29b94b97a6037b9f890ba291 (127.15.250.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793cd2aa95144e30aaa7a21ca56039c2" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 44453 } } peers { permanent_uuid: "5593115523c94df5bb97f3c3a53dc5b5" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34957 } } peers { permanent_uuid: "675705f02b8044a28cd6f540ecba998a" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 42869 } } }
I20260808 07:00:51.526533 16407 catalog_manager.cc:5495] T 9cdab77050924e1ca1cec06b14f932a6 P 675705f02b8044a28cd6f540ecba998a reported cstate change: config changed from index 2 to 3, NON_VOTER b51ae32d29b94b97a6037b9f890ba291 (127.15.250.132) evicted. New cstate: current_term: 1 leader_uuid: "5593115523c94df5bb97f3c3a53dc5b5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793cd2aa95144e30aaa7a21ca56039c2" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 44453 } } peers { permanent_uuid: "5593115523c94df5bb97f3c3a53dc5b5" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34957 } } peers { permanent_uuid: "675705f02b8044a28cd6f540ecba998a" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 42869 } } }
I20260808 07:00:51.526624 16392 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 9cdab77050924e1ca1cec06b14f932a6 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 07:00:51.530488 16984 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cdab77050924e1ca1cec06b14f932a6 with delete_type TABLET_DATA_TOMBSTONED (TS b51ae32d29b94b97a6037b9f890ba291 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:49500
W20260808 07:00:51.531320 16391 catalog_manager.cc:4808] TS b51ae32d29b94b97a6037b9f890ba291 (127.15.250.132:38085): delete failed for tablet 9cdab77050924e1ca1cec06b14f932a6 because the tablet was not found. No further retry: Not found: Tablet not found: 9cdab77050924e1ca1cec06b14f932a6
W20260808 07:00:51.566794 16734 generic_service.cc:139] {username='slave'} at 127.0.0.1:33052 forcing change of non-runtime-safe flag fail_dns_resolution
I20260808 07:00:51.566884 16734 generic_service.cc:175] {username='slave'} at 127.0.0.1:33052 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
W20260808 07:00:51.567395 16872 generic_service.cc:139] {username='slave'} at 127.0.0.1:58782 forcing change of non-runtime-safe flag fail_dns_resolution
I20260808 07:00:51.567461 16872 generic_service.cc:175] {username='slave'} at 127.0.0.1:58782 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
W20260808 07:00:51.567888 16596 generic_service.cc:139] {username='slave'} at 127.0.0.1:55064 forcing change of non-runtime-safe flag fail_dns_resolution
I20260808 07:00:51.567947 16596 generic_service.cc:175] {username='slave'} at 127.0.0.1:55064 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
I20260808 07:00:51.568557 16829 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9cdab77050924e1ca1cec06b14f932a6"
dest_uuid: "5593115523c94df5bb97f3c3a53dc5b5"
mode: GRACEFUL
new_leader_uuid: "5593115523c94df5bb97f3c3a53dc5b5"
from {username='slave'} at 127.0.0.1:58774
I20260808 07:00:51.568648 16829 raft_consensus.cc:595] T 9cdab77050924e1ca1cec06b14f932a6 P 5593115523c94df5bb97f3c3a53dc5b5 [term 1 LEADER]: Received request to transfer leadership to TS 5593115523c94df5bb97f3c3a53dc5b5
I20260808 07:00:52.518217 17053 heartbeater.cc:503] Master 127.15.250.190:40665 was elected leader, sending a full tablet report...
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/raft_consensus_election-itest.cc:292: Failure
Value of: tablets.empty()
Actual: true
Expected: false
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_util.cc:395: Failure
Failed
Timed out waiting for assertion to pass.
I20260808 07:01:21.625340 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 16470
I20260808 07:01:21.631290 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 16605
I20260808 07:01:21.636708 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 16745
I20260808 07:01:21.642659 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 16919
I20260808 07:01:21.647545 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 16374
2026-08-08T07:01:21Z chronyd exiting
I20260808 07:01:21.665206 16362 test_util.cc:175] -----------------------------------------------
I20260808 07:01:21.665292 16362 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1786172446491335-16362-0
[ FAILED ] RaftConsensusElectionITest.TestNewLeaderCantResolvePeers (35162 ms)
[ RUN ] RaftConsensusElectionITest.RunLeaderElection
I20260808 07:01:21.665835 16362 test_util.cc:268] Using random seed: -1337525464
I20260808 07:01:21.666399 16362 ts_itest-base.cc:112] Starting cluster with:
I20260808 07:01:21.666448 16362 ts_itest-base.cc:113] --------------
I20260808 07:01:21.666467 16362 ts_itest-base.cc:114] 3 tablet servers
I20260808 07:01:21.666496 16362 ts_itest-base.cc:115] 3 replicas per TS
I20260808 07:01:21.666520 16362 ts_itest-base.cc:116] --------------
2026-08-08T07:01:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:01:21Z Disabled control of system clock
I20260808 07:01:21.675526 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.250.190:44153
--webserver_interface=127.15.250.190
--webserver_port=0
--builtin_ntp_servers=127.15.250.148:42217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.250.190:44153 with env {}
W20260808 07:01:21.761797 17078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:21.762055 17078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:21.762089 17078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:21.763288 17078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:01:21.763355 17078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:21.763381 17078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:01:21.763413 17078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:01:21.764912 17078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:42217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.250.190:44153
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.250.190:44153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.250.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.17078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:01:21.765254 17078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:21.765615 17078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:21.770805 17087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:21.770839 17085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:21.770829 17084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:21.771201 17078 server_base.cc:1034] running on GCE node
I20260808 07:01:21.771451 17078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:21.771795 17078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:21.772943 17078 hybrid_clock.cc:648] HybridClock initialized: now 1786172481772922 us; error 41 us; skew 500 ppm
I20260808 07:01:21.777547 17078 webserver.cc:460] Webserver started at http://127.15.250.190:33937/ using document root <none> and password file <none>
I20260808 07:01:21.777840 17078 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:21.777918 17078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:21.778100 17078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:21.779206 17078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "e0d1eb4f77554f4c8b3cb4c586625a7a"
format_stamp: "Formatted at 2026-08-08 07:01:21 on dist-test-slave-0b58"
I20260808 07:01:21.779592 17078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "e0d1eb4f77554f4c8b3cb4c586625a7a"
format_stamp: "Formatted at 2026-08-08 07:01:21 on dist-test-slave-0b58"
I20260808 07:01:21.781247 17078 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 07:01:21.782141 17093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:21.782403 17078 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:01:21.782528 17078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e0d1eb4f77554f4c8b3cb4c586625a7a"
format_stamp: "Formatted at 2026-08-08 07:01:21 on dist-test-slave-0b58"
I20260808 07:01:21.782616 17078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:21.816238 17078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:21.816701 17078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:21.816875 17078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:21.821081 17078 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.190:44153
I20260808 07:01:21.821118 17149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.190:44153 every 8 connection(s)
I20260808 07:01:21.821524 17078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260808 07:01:21.822218 17150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:21.824965 17150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0d1eb4f77554f4c8b3cb4c586625a7a: Bootstrap starting.
I20260808 07:01:21.825558 17150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0d1eb4f77554f4c8b3cb4c586625a7a: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:21.825819 17150 log.cc:824] T 00000000000000000000000000000000 P e0d1eb4f77554f4c8b3cb4c586625a7a: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:21.826483 17150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0d1eb4f77554f4c8b3cb4c586625a7a: No bootstrap required, opened a new log
I20260808 07:01:21.827993 17150 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0d1eb4f77554f4c8b3cb4c586625a7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d1eb4f77554f4c8b3cb4c586625a7a" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 44153 } }
I20260808 07:01:21.828135 17150 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0d1eb4f77554f4c8b3cb4c586625a7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:01:21.828156 17150 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0d1eb4f77554f4c8b3cb4c586625a7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0d1eb4f77554f4c8b3cb4c586625a7a, State: Initialized, Role: FOLLOWER
I20260808 07:01:21.828310 17150 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0d1eb4f77554f4c8b3cb4c586625a7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d1eb4f77554f4c8b3cb4c586625a7a" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 44153 } }
I20260808 07:01:21.828392 17150 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0d1eb4f77554f4c8b3cb4c586625a7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:01:21.828414 17150 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0d1eb4f77554f4c8b3cb4c586625a7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:01:21.828444 17150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0d1eb4f77554f4c8b3cb4c586625a7a [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:01:21.829001 17150 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0d1eb4f77554f4c8b3cb4c586625a7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d1eb4f77554f4c8b3cb4c586625a7a" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 44153 } }
I20260808 07:01:21.829090 17150 leader_election.cc:302] T 00000000000000000000000000000000 P e0d1eb4f77554f4c8b3cb4c586625a7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0d1eb4f77554f4c8b3cb4c586625a7a; no voters:
I20260808 07:01:21.829213 17150 leader_election.cc:288] T 00000000000000000000000000000000 P e0d1eb4f77554f4c8b3cb4c586625a7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:01:21.829344 17153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0d1eb4f77554f4c8b3cb4c586625a7a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:01:21.829501 17153 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0d1eb4f77554f4c8b3cb4c586625a7a [term 1 LEADER]: Becoming Leader. State: Replica: e0d1eb4f77554f4c8b3cb4c586625a7a, State: Running, Role: LEADER
I20260808 07:01:21.829598 17153 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0d1eb4f77554f4c8b3cb4c586625a7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d1eb4f77554f4c8b3cb4c586625a7a" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 44153 } }
I20260808 07:01:21.829511 17150 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0d1eb4f77554f4c8b3cb4c586625a7a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:01:21.830006 17155 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0d1eb4f77554f4c8b3cb4c586625a7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0d1eb4f77554f4c8b3cb4c586625a7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d1eb4f77554f4c8b3cb4c586625a7a" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 44153 } } }
I20260808 07:01:21.830044 17156 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0d1eb4f77554f4c8b3cb4c586625a7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0d1eb4f77554f4c8b3cb4c586625a7a. Latest consensus state: current_term: 1 leader_uuid: "e0d1eb4f77554f4c8b3cb4c586625a7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d1eb4f77554f4c8b3cb4c586625a7a" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 44153 } } }
I20260808 07:01:21.830101 17155 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0d1eb4f77554f4c8b3cb4c586625a7a [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:21.830124 17156 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0d1eb4f77554f4c8b3cb4c586625a7a [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:21.830327 17160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:01:21.831156 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 17078
I20260808 07:01:21.831243 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260808 07:01:21.831277 17160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:01:21.836383 17160 catalog_manager.cc:1347] Generated new cluster ID: c4bd6aa0c62e400caac4a4a63a11703d
I20260808 07:01:21.836457 17160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:01:21.854202 17160 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:01:21.854790 17160 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:01:21.892632 17160 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0d1eb4f77554f4c8b3cb4c586625a7a: Generated new TSK 0
I20260808 07:01:21.892872 17160 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:01:21.903946 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.129:0
--local_ip_for_outbound_sockets=127.15.250.129
--webserver_interface=127.15.250.129
--webserver_port=0
--tserver_master_addrs=127.15.250.190:44153
--builtin_ntp_servers=127.15.250.148:42217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
W20260808 07:01:22.009603 17176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:22.009826 17176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:22.009877 17176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:22.011111 17176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:22.011168 17176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.129
I20260808 07:01:22.012566 17176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:42217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.250.129
--webserver_port=0
--tserver_master_addrs=127.15.250.190:44153
--never_fsync=true
--heap_profile_path=/tmp/kudu.17176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.129
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:01:22.012771 17176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:22.013036 17176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:22.018302 17184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:22.018520 17176 server_base.cc:1034] running on GCE node
W20260808 07:01:22.018302 17181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:22.018306 17182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:22.018867 17176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:22.019101 17176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:22.020241 17176 hybrid_clock.cc:648] HybridClock initialized: now 1786172482020217 us; error 48 us; skew 500 ppm
I20260808 07:01:22.021592 17176 webserver.cc:460] Webserver started at http://127.15.250.129:46407/ using document root <none> and password file <none>
I20260808 07:01:22.021871 17176 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:22.021945 17176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:22.022104 17176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:22.023152 17176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "49c4b16a457f45f28bef76b6a2b42faa"
format_stamp: "Formatted at 2026-08-08 07:01:22 on dist-test-slave-0b58"
I20260808 07:01:22.024518 17176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "49c4b16a457f45f28bef76b6a2b42faa"
format_stamp: "Formatted at 2026-08-08 07:01:22 on dist-test-slave-0b58"
I20260808 07:01:22.025952 17176 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 07:01:22.026769 17191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:22.026973 17176 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:01:22.027081 17176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "49c4b16a457f45f28bef76b6a2b42faa"
format_stamp: "Formatted at 2026-08-08 07:01:22 on dist-test-slave-0b58"
I20260808 07:01:22.027159 17176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:22.036835 17176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:22.037156 17176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:22.037308 17176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:22.037539 17176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:01:22.037891 17176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:01:22.037945 17176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:22.037981 17176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:01:22.038007 17176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:22.043751 17176 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.129:37955
I20260808 07:01:22.043836 17309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.129:37955 every 8 connection(s)
I20260808 07:01:22.044179 17176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260808 07:01:22.050194 17310 heartbeater.cc:347] Connected to a master server at 127.15.250.190:44153
I20260808 07:01:22.050289 17310 heartbeater.cc:464] Registering TS with master...
I20260808 07:01:22.050486 17310 heartbeater.cc:511] Master 127.15.250.190:44153 requested a full tablet report, sending...
I20260808 07:01:22.050714 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 17176
I20260808 07:01:22.050851 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260808 07:01:22.051085 17112 ts_manager.cc:194] Registered new tserver with Master: 49c4b16a457f45f28bef76b6a2b42faa (127.15.250.129:37955)
I20260808 07:01:22.052134 17112 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.129:47161
I20260808 07:01:22.052318 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.130:0
--local_ip_for_outbound_sockets=127.15.250.130
--webserver_interface=127.15.250.130
--webserver_port=0
--tserver_master_addrs=127.15.250.190:44153
--builtin_ntp_servers=127.15.250.148:42217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
W20260808 07:01:22.139269 17313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:22.139531 17313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:22.139570 17313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:22.140923 17313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:22.141012 17313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.130
I20260808 07:01:22.142396 17313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:42217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.250.130
--webserver_port=0
--tserver_master_addrs=127.15.250.190:44153
--never_fsync=true
--heap_profile_path=/tmp/kudu.17313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.130
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:01:22.142702 17313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:22.143045 17313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:22.148173 17318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:22.148176 17322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:22.148178 17319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:22.148495 17313 server_base.cc:1034] running on GCE node
I20260808 07:01:22.148726 17313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:22.148989 17313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:22.150187 17313 hybrid_clock.cc:648] HybridClock initialized: now 1786172482150176 us; error 30 us; skew 500 ppm
I20260808 07:01:22.151677 17313 webserver.cc:460] Webserver started at http://127.15.250.130:35907/ using document root <none> and password file <none>
I20260808 07:01:22.151988 17313 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:22.152040 17313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:22.152145 17313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:22.153208 17313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "951cdf4e3e67487da6c4eab75ed97eb4"
format_stamp: "Formatted at 2026-08-08 07:01:22 on dist-test-slave-0b58"
I20260808 07:01:22.153522 17313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "951cdf4e3e67487da6c4eab75ed97eb4"
format_stamp: "Formatted at 2026-08-08 07:01:22 on dist-test-slave-0b58"
I20260808 07:01:22.154882 17313 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 07:01:22.155707 17328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:22.155956 17313 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 07:01:22.156082 17313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "951cdf4e3e67487da6c4eab75ed97eb4"
format_stamp: "Formatted at 2026-08-08 07:01:22 on dist-test-slave-0b58"
I20260808 07:01:22.156225 17313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:22.174803 17313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:22.175182 17313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:22.175369 17313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:22.175647 17313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:01:22.176043 17313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:01:22.176105 17313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:22.176146 17313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:01:22.176203 17313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:22.182628 17313 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.130:33901
I20260808 07:01:22.182672 17448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.130:33901 every 8 connection(s)
I20260808 07:01:22.183064 17313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260808 07:01:22.188284 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 17313
I20260808 07:01:22.188427 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260808 07:01:22.189563 17449 heartbeater.cc:347] Connected to a master server at 127.15.250.190:44153
I20260808 07:01:22.189687 17449 heartbeater.cc:464] Registering TS with master...
I20260808 07:01:22.189808 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.131:0
--local_ip_for_outbound_sockets=127.15.250.131
--webserver_interface=127.15.250.131
--webserver_port=0
--tserver_master_addrs=127.15.250.190:44153
--builtin_ntp_servers=127.15.250.148:42217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
I20260808 07:01:22.189945 17449 heartbeater.cc:511] Master 127.15.250.190:44153 requested a full tablet report, sending...
I20260808 07:01:22.190374 17112 ts_manager.cc:194] Registered new tserver with Master: 951cdf4e3e67487da6c4eab75ed97eb4 (127.15.250.130:33901)
I20260808 07:01:22.190896 17112 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.130:44639
W20260808 07:01:22.276588 17452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:22.276809 17452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:22.276831 17452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:22.278007 17452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:22.278059 17452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.131
I20260808 07:01:22.279402 17452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:42217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.250.131
--webserver_port=0
--tserver_master_addrs=127.15.250.190:44153
--never_fsync=true
--heap_profile_path=/tmp/kudu.17452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.131
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:01:22.279608 17452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:22.279876 17452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:22.285153 17457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:22.285108 17460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:22.285115 17458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:22.285442 17452 server_base.cc:1034] running on GCE node
I20260808 07:01:22.285702 17452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:22.285948 17452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:22.287115 17452 hybrid_clock.cc:648] HybridClock initialized: now 1786172482287103 us; error 48 us; skew 500 ppm
I20260808 07:01:22.288583 17452 webserver.cc:460] Webserver started at http://127.15.250.131:33575/ using document root <none> and password file <none>
I20260808 07:01:22.288859 17452 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:22.288936 17452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:22.289089 17452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:22.290086 17452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "5abd0c1ef2ba4a019b905aa559b26c93"
format_stamp: "Formatted at 2026-08-08 07:01:22 on dist-test-slave-0b58"
I20260808 07:01:22.290464 17452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "5abd0c1ef2ba4a019b905aa559b26c93"
format_stamp: "Formatted at 2026-08-08 07:01:22 on dist-test-slave-0b58"
I20260808 07:01:22.291819 17452 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 07:01:22.292608 17466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:22.292793 17452 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:22.292903 17452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "5abd0c1ef2ba4a019b905aa559b26c93"
format_stamp: "Formatted at 2026-08-08 07:01:22 on dist-test-slave-0b58"
I20260808 07:01:22.292986 17452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:22.308353 17452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:22.308684 17452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:22.308840 17452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:22.309069 17452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:01:22.309464 17452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:01:22.309522 17452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:22.309559 17452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:01:22.309587 17452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:22.315272 17452 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.131:33161
I20260808 07:01:22.315313 17586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.131:33161 every 8 connection(s)
I20260808 07:01:22.315660 17452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260808 07:01:22.315801 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 17452
I20260808 07:01:22.315909 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260808 07:01:22.321578 17587 heartbeater.cc:347] Connected to a master server at 127.15.250.190:44153
I20260808 07:01:22.321671 17587 heartbeater.cc:464] Registering TS with master...
I20260808 07:01:22.321828 17587 heartbeater.cc:511] Master 127.15.250.190:44153 requested a full tablet report, sending...
I20260808 07:01:22.322151 17112 ts_manager.cc:194] Registered new tserver with Master: 5abd0c1ef2ba4a019b905aa559b26c93 (127.15.250.131:33161)
I20260808 07:01:22.322538 17112 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.131:44529
I20260808 07:01:22.328019 16362 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 07:01:22.335001 17112 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43912:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20260808 07:01:22.336620 17112 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 07:01:22.343796 17239 tablet_service.cc:1467] Processing CreateTablet for tablet 3dce9bd5194b4c3ea05360f1512e3cdd (DEFAULT_TABLE table=TestTable [id=693a88b7bef34fa9bdc187feb8f42c08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:01:22.343945 17379 tablet_service.cc:1467] Processing CreateTablet for tablet 3dce9bd5194b4c3ea05360f1512e3cdd (DEFAULT_TABLE table=TestTable [id=693a88b7bef34fa9bdc187feb8f42c08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:01:22.343946 17519 tablet_service.cc:1467] Processing CreateTablet for tablet 3dce9bd5194b4c3ea05360f1512e3cdd (DEFAULT_TABLE table=TestTable [id=693a88b7bef34fa9bdc187feb8f42c08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:01:22.344195 17239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dce9bd5194b4c3ea05360f1512e3cdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:22.344331 17519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dce9bd5194b4c3ea05360f1512e3cdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:22.344331 17379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dce9bd5194b4c3ea05360f1512e3cdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:22.346868 17601 tablet_bootstrap.cc:492] T 3dce9bd5194b4c3ea05360f1512e3cdd P 49c4b16a457f45f28bef76b6a2b42faa: Bootstrap starting.
I20260808 07:01:22.347504 17603 tablet_bootstrap.cc:492] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4: Bootstrap starting.
I20260808 07:01:22.347929 17601 tablet_bootstrap.cc:654] T 3dce9bd5194b4c3ea05360f1512e3cdd P 49c4b16a457f45f28bef76b6a2b42faa: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:22.348182 17603 tablet_bootstrap.cc:654] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:22.348290 17601 log.cc:824] T 3dce9bd5194b4c3ea05360f1512e3cdd P 49c4b16a457f45f28bef76b6a2b42faa: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:22.348425 17602 tablet_bootstrap.cc:492] T 3dce9bd5194b4c3ea05360f1512e3cdd P 5abd0c1ef2ba4a019b905aa559b26c93: Bootstrap starting.
I20260808 07:01:22.348570 17603 log.cc:824] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:22.349105 17601 tablet_bootstrap.cc:492] T 3dce9bd5194b4c3ea05360f1512e3cdd P 49c4b16a457f45f28bef76b6a2b42faa: No bootstrap required, opened a new log
I20260808 07:01:22.349134 17602 tablet_bootstrap.cc:654] T 3dce9bd5194b4c3ea05360f1512e3cdd P 5abd0c1ef2ba4a019b905aa559b26c93: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:22.349170 17603 tablet_bootstrap.cc:492] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4: No bootstrap required, opened a new log
I20260808 07:01:22.349181 17601 ts_tablet_manager.cc:1397] T 3dce9bd5194b4c3ea05360f1512e3cdd P 49c4b16a457f45f28bef76b6a2b42faa: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 07:01:22.349215 17603 ts_tablet_manager.cc:1397] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 07:01:22.349472 17602 log.cc:824] T 3dce9bd5194b4c3ea05360f1512e3cdd P 5abd0c1ef2ba4a019b905aa559b26c93: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:22.350204 17602 tablet_bootstrap.cc:492] T 3dce9bd5194b4c3ea05360f1512e3cdd P 5abd0c1ef2ba4a019b905aa559b26c93: No bootstrap required, opened a new log
I20260808 07:01:22.350270 17602 ts_tablet_manager.cc:1397] T 3dce9bd5194b4c3ea05360f1512e3cdd P 5abd0c1ef2ba4a019b905aa559b26c93: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:01:22.351118 17603 raft_consensus.cc:348] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c4b16a457f45f28bef76b6a2b42faa" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 37955 } } peers { permanent_uuid: "5abd0c1ef2ba4a019b905aa559b26c93" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 33161 } } peers { permanent_uuid: "951cdf4e3e67487da6c4eab75ed97eb4" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 33901 } }
I20260808 07:01:22.351253 17603 raft_consensus.cc:374] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:01:22.351275 17603 raft_consensus.cc:729] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951cdf4e3e67487da6c4eab75ed97eb4, State: Initialized, Role: FOLLOWER
I20260808 07:01:22.351398 17603 consensus_queue.cc:260] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c4b16a457f45f28bef76b6a2b42faa" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 37955 } } peers { permanent_uuid: "5abd0c1ef2ba4a019b905aa559b26c93" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 33161 } } peers { permanent_uuid: "951cdf4e3e67487da6c4eab75ed97eb4" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 33901 } }
I20260808 07:01:22.351687 17603 ts_tablet_manager.cc:1428] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260808 07:01:22.351815 17449 heartbeater.cc:503] Master 127.15.250.190:44153 was elected leader, sending a full tablet report...
I20260808 07:01:22.351924 17601 raft_consensus.cc:348] T 3dce9bd5194b4c3ea05360f1512e3cdd P 49c4b16a457f45f28bef76b6a2b42faa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c4b16a457f45f28bef76b6a2b42faa" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 37955 } } peers { permanent_uuid: "5abd0c1ef2ba4a019b905aa559b26c93" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 33161 } } peers { permanent_uuid: "951cdf4e3e67487da6c4eab75ed97eb4" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 33901 } }
I20260808 07:01:22.352087 17601 raft_consensus.cc:374] T 3dce9bd5194b4c3ea05360f1512e3cdd P 49c4b16a457f45f28bef76b6a2b42faa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:01:22.352123 17601 raft_consensus.cc:729] T 3dce9bd5194b4c3ea05360f1512e3cdd P 49c4b16a457f45f28bef76b6a2b42faa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c4b16a457f45f28bef76b6a2b42faa, State: Initialized, Role: FOLLOWER
I20260808 07:01:22.352226 17601 consensus_queue.cc:260] T 3dce9bd5194b4c3ea05360f1512e3cdd P 49c4b16a457f45f28bef76b6a2b42faa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c4b16a457f45f28bef76b6a2b42faa" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 37955 } } peers { permanent_uuid: "5abd0c1ef2ba4a019b905aa559b26c93" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 33161 } } peers { permanent_uuid: "951cdf4e3e67487da6c4eab75ed97eb4" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 33901 } }
I20260808 07:01:22.352545 17601 ts_tablet_manager.cc:1428] T 3dce9bd5194b4c3ea05360f1512e3cdd P 49c4b16a457f45f28bef76b6a2b42faa: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260808 07:01:22.352752 17310 heartbeater.cc:503] Master 127.15.250.190:44153 was elected leader, sending a full tablet report...
I20260808 07:01:22.353161 17602 raft_consensus.cc:348] T 3dce9bd5194b4c3ea05360f1512e3cdd P 5abd0c1ef2ba4a019b905aa559b26c93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c4b16a457f45f28bef76b6a2b42faa" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 37955 } } peers { permanent_uuid: "5abd0c1ef2ba4a019b905aa559b26c93" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 33161 } } peers { permanent_uuid: "951cdf4e3e67487da6c4eab75ed97eb4" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 33901 } }
I20260808 07:01:22.353335 17602 raft_consensus.cc:374] T 3dce9bd5194b4c3ea05360f1512e3cdd P 5abd0c1ef2ba4a019b905aa559b26c93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:01:22.353377 17602 raft_consensus.cc:729] T 3dce9bd5194b4c3ea05360f1512e3cdd P 5abd0c1ef2ba4a019b905aa559b26c93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5abd0c1ef2ba4a019b905aa559b26c93, State: Initialized, Role: FOLLOWER
I20260808 07:01:22.353492 17602 consensus_queue.cc:260] T 3dce9bd5194b4c3ea05360f1512e3cdd P 5abd0c1ef2ba4a019b905aa559b26c93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c4b16a457f45f28bef76b6a2b42faa" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 37955 } } peers { permanent_uuid: "5abd0c1ef2ba4a019b905aa559b26c93" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 33161 } } peers { permanent_uuid: "951cdf4e3e67487da6c4eab75ed97eb4" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 33901 } }
I20260808 07:01:22.353754 17602 ts_tablet_manager.cc:1428] T 3dce9bd5194b4c3ea05360f1512e3cdd P 5abd0c1ef2ba4a019b905aa559b26c93: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260808 07:01:22.353767 17587 heartbeater.cc:503] Master 127.15.250.190:44153 was elected leader, sending a full tablet report...
W20260808 07:01:22.433744 17450 tablet.cc:2369] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 07:01:22.544888 17311 tablet.cc:2369] T 3dce9bd5194b4c3ea05360f1512e3cdd P 49c4b16a457f45f28bef76b6a2b42faa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 07:01:22.566319 17588 tablet.cc:2369] T 3dce9bd5194b4c3ea05360f1512e3cdd P 5abd0c1ef2ba4a019b905aa559b26c93: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 07:01:22.706445 17607 raft_consensus.cc:482] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 07:01:22.706630 17607 raft_consensus.cc:504] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c4b16a457f45f28bef76b6a2b42faa" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 37955 } } peers { permanent_uuid: "5abd0c1ef2ba4a019b905aa559b26c93" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 33161 } } peers { permanent_uuid: "951cdf4e3e67487da6c4eab75ed97eb4" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 33901 } }
I20260808 07:01:22.707002 17607 leader_election.cc:288] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49c4b16a457f45f28bef76b6a2b42faa (127.15.250.129:37955), 5abd0c1ef2ba4a019b905aa559b26c93 (127.15.250.131:33161)
I20260808 07:01:22.710403 17539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dce9bd5194b4c3ea05360f1512e3cdd" candidate_uuid: "951cdf4e3e67487da6c4eab75ed97eb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5abd0c1ef2ba4a019b905aa559b26c93" is_pre_election: true
I20260808 07:01:22.710443 17261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dce9bd5194b4c3ea05360f1512e3cdd" candidate_uuid: "951cdf4e3e67487da6c4eab75ed97eb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c4b16a457f45f28bef76b6a2b42faa" is_pre_election: true
I20260808 07:01:22.710590 17261 raft_consensus.cc:2454] T 3dce9bd5194b4c3ea05360f1512e3cdd P 49c4b16a457f45f28bef76b6a2b42faa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 951cdf4e3e67487da6c4eab75ed97eb4 in term 0.
I20260808 07:01:22.710592 17539 raft_consensus.cc:2454] T 3dce9bd5194b4c3ea05360f1512e3cdd P 5abd0c1ef2ba4a019b905aa559b26c93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 951cdf4e3e67487da6c4eab75ed97eb4 in term 0.
I20260808 07:01:22.710886 17332 leader_election.cc:302] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5abd0c1ef2ba4a019b905aa559b26c93, 951cdf4e3e67487da6c4eab75ed97eb4; no voters:
I20260808 07:01:22.711090 17607 raft_consensus.cc:2781] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 07:01:22.711135 17607 raft_consensus.cc:482] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 07:01:22.711158 17607 raft_consensus.cc:3037] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:01:22.711815 17607 raft_consensus.cc:504] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c4b16a457f45f28bef76b6a2b42faa" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 37955 } } peers { permanent_uuid: "5abd0c1ef2ba4a019b905aa559b26c93" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 33161 } } peers { permanent_uuid: "951cdf4e3e67487da6c4eab75ed97eb4" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 33901 } }
I20260808 07:01:22.711970 17607 leader_election.cc:288] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4 [CANDIDATE]: Term 1 election: Requested vote from peers 49c4b16a457f45f28bef76b6a2b42faa (127.15.250.129:37955), 5abd0c1ef2ba4a019b905aa559b26c93 (127.15.250.131:33161)
I20260808 07:01:22.712136 17539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dce9bd5194b4c3ea05360f1512e3cdd" candidate_uuid: "951cdf4e3e67487da6c4eab75ed97eb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5abd0c1ef2ba4a019b905aa559b26c93"
I20260808 07:01:22.712136 17261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dce9bd5194b4c3ea05360f1512e3cdd" candidate_uuid: "951cdf4e3e67487da6c4eab75ed97eb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c4b16a457f45f28bef76b6a2b42faa"
I20260808 07:01:22.712229 17539 raft_consensus.cc:3037] T 3dce9bd5194b4c3ea05360f1512e3cdd P 5abd0c1ef2ba4a019b905aa559b26c93 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:01:22.712229 17261 raft_consensus.cc:3037] T 3dce9bd5194b4c3ea05360f1512e3cdd P 49c4b16a457f45f28bef76b6a2b42faa [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:01:22.713176 17261 raft_consensus.cc:2454] T 3dce9bd5194b4c3ea05360f1512e3cdd P 49c4b16a457f45f28bef76b6a2b42faa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 951cdf4e3e67487da6c4eab75ed97eb4 in term 1.
I20260808 07:01:22.713176 17539 raft_consensus.cc:2454] T 3dce9bd5194b4c3ea05360f1512e3cdd P 5abd0c1ef2ba4a019b905aa559b26c93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 951cdf4e3e67487da6c4eab75ed97eb4 in term 1.
I20260808 07:01:22.713370 17331 leader_election.cc:302] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c4b16a457f45f28bef76b6a2b42faa, 951cdf4e3e67487da6c4eab75ed97eb4; no voters:
I20260808 07:01:22.713511 17607 raft_consensus.cc:2781] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:01:22.713696 17607 raft_consensus.cc:686] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4 [term 1 LEADER]: Becoming Leader. State: Replica: 951cdf4e3e67487da6c4eab75ed97eb4, State: Running, Role: LEADER
I20260808 07:01:22.713812 17607 consensus_queue.cc:237] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c4b16a457f45f28bef76b6a2b42faa" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 37955 } } peers { permanent_uuid: "5abd0c1ef2ba4a019b905aa559b26c93" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 33161 } } peers { permanent_uuid: "951cdf4e3e67487da6c4eab75ed97eb4" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 33901 } }
I20260808 07:01:22.714709 17111 catalog_manager.cc:5495] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 951cdf4e3e67487da6c4eab75ed97eb4 (127.15.250.130). New cstate: current_term: 1 leader_uuid: "951cdf4e3e67487da6c4eab75ed97eb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c4b16a457f45f28bef76b6a2b42faa" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 37955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5abd0c1ef2ba4a019b905aa559b26c93" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 33161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "951cdf4e3e67487da6c4eab75ed97eb4" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 33901 } health_report { overall_health: HEALTHY } } }
I20260808 07:01:22.786693 16362 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 07:01:22.787470 16362 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 49c4b16a457f45f28bef76b6a2b42faa to finish bootstrapping
I20260808 07:01:22.791770 16362 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 951cdf4e3e67487da6c4eab75ed97eb4 to finish bootstrapping
I20260808 07:01:22.795680 16362 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 5abd0c1ef2ba4a019b905aa559b26c93 to finish bootstrapping
I20260808 07:01:22.813252 17539 raft_consensus.cc:1261] T 3dce9bd5194b4c3ea05360f1512e3cdd P 5abd0c1ef2ba4a019b905aa559b26c93 [term 1 FOLLOWER]: Refusing update from remote peer 951cdf4e3e67487da6c4eab75ed97eb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 07:01:22.813278 17261 raft_consensus.cc:1261] T 3dce9bd5194b4c3ea05360f1512e3cdd P 49c4b16a457f45f28bef76b6a2b42faa [term 1 FOLLOWER]: Refusing update from remote peer 951cdf4e3e67487da6c4eab75ed97eb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 07:01:22.813694 17607 consensus_queue.cc:1035] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c4b16a457f45f28bef76b6a2b42faa" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 37955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:01:22.813920 17612 consensus_queue.cc:1035] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5abd0c1ef2ba4a019b905aa559b26c93" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 33161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:01:22.816867 17618 mvcc.cc:205] Tried to move back new op lower bound from 7316162489600200704 to 7316162489196400640. Current Snapshot: MvccSnapshot[applied={T|T < 7316162489600200704}]
I20260808 07:01:22.818867 17620 mvcc.cc:205] Tried to move back new op lower bound from 7316162489600200704 to 7316162489196400640. Current Snapshot: MvccSnapshot[applied={T|T < 7316162489600200704}]
I20260808 07:01:22.855893 17379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260808 07:01:22.855911 17519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260808 07:01:22.856164 17239 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e0d1eb4f77554f4c8b3cb4c586625a7a | 127.15.250.190:44153 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.250.148:42217 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
49c4b16a457f45f28bef76b6a2b42faa | 127.15.250.129:37955 | HEALTHY | <none> | 0 | 0
5abd0c1ef2ba4a019b905aa559b26c93 | 127.15.250.131:33161 | HEALTHY | <none> | 0 | 0
951cdf4e3e67487da6c4eab75ed97eb4 | 127.15.250.130:33901 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.15.250.129 | experimental | 127.15.250.129:37955
local_ip_for_outbound_sockets | 127.15.250.130 | experimental | 127.15.250.130:33901
local_ip_for_outbound_sockets | 127.15.250.131 | experimental | 127.15.250.131:33161
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.15.250.129:37955
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.15.250.130:33901
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.15.250.131:33161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.250.148:42217 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260808 07:01:22.872547 16362 log_verifier.cc:126] Checking tablet 3dce9bd5194b4c3ea05360f1512e3cdd
I20260808 07:01:22.895742 16362 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 3dce9bd5194b4c3ea05360f1512e3cdd
W20260808 07:01:23.432117 16362 scanner-internal.cc:458] Time spent opening tablet: real 0.529s user 0.000s sys 0.001s
I20260808 07:01:23.445319 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 17313
I20260808 07:01:23.454743 17261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3dce9bd5194b4c3ea05360f1512e3cdd"
dest_uuid: "49c4b16a457f45f28bef76b6a2b42faa"
from {username='slave'} at 127.0.0.1:43008
I20260808 07:01:23.454907 17261 raft_consensus.cc:482] T 3dce9bd5194b4c3ea05360f1512e3cdd P 49c4b16a457f45f28bef76b6a2b42faa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 07:01:23.454957 17261 raft_consensus.cc:3037] T 3dce9bd5194b4c3ea05360f1512e3cdd P 49c4b16a457f45f28bef76b6a2b42faa [term 1 FOLLOWER]: Advancing to term 2
I20260808 07:01:23.455633 17261 raft_consensus.cc:504] T 3dce9bd5194b4c3ea05360f1512e3cdd P 49c4b16a457f45f28bef76b6a2b42faa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c4b16a457f45f28bef76b6a2b42faa" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 37955 } } peers { permanent_uuid: "5abd0c1ef2ba4a019b905aa559b26c93" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 33161 } } peers { permanent_uuid: "951cdf4e3e67487da6c4eab75ed97eb4" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 33901 } }
I20260808 07:01:23.455979 17261 leader_election.cc:288] T 3dce9bd5194b4c3ea05360f1512e3cdd P 49c4b16a457f45f28bef76b6a2b42faa [CANDIDATE]: Term 2 election: Requested vote from peers 5abd0c1ef2ba4a019b905aa559b26c93 (127.15.250.131:33161), 951cdf4e3e67487da6c4eab75ed97eb4 (127.15.250.130:33901)
W20260808 07:01:23.456390 17193 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.250.130:33901: connect: Connection refused (error 111)
W20260808 07:01:23.456847 17193 leader_election.cc:334] T 3dce9bd5194b4c3ea05360f1512e3cdd P 49c4b16a457f45f28bef76b6a2b42faa [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 951cdf4e3e67487da6c4eab75ed97eb4 (127.15.250.130:33901): Network error: Client connection negotiation failed: client connection to 127.15.250.130:33901: connect: Connection refused (error 111)
W20260808 07:01:23.458336 17591 meta_cache.cc:302] tablet 3dce9bd5194b4c3ea05360f1512e3cdd: replica 951cdf4e3e67487da6c4eab75ed97eb4 (127.15.250.130:33901) has failed: Network error: Client connection negotiation failed: client connection to 127.15.250.130:33901: connect: Connection refused (error 111)
I20260808 07:01:23.459614 17539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dce9bd5194b4c3ea05360f1512e3cdd" candidate_uuid: "49c4b16a457f45f28bef76b6a2b42faa" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "5abd0c1ef2ba4a019b905aa559b26c93"
I20260808 07:01:23.459733 17539 raft_consensus.cc:3037] T 3dce9bd5194b4c3ea05360f1512e3cdd P 5abd0c1ef2ba4a019b905aa559b26c93 [term 1 FOLLOWER]: Advancing to term 2
I20260808 07:01:23.460500 17539 raft_consensus.cc:2454] T 3dce9bd5194b4c3ea05360f1512e3cdd P 5abd0c1ef2ba4a019b905aa559b26c93 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c4b16a457f45f28bef76b6a2b42faa in term 2.
I20260808 07:01:23.460721 17195 leader_election.cc:302] T 3dce9bd5194b4c3ea05360f1512e3cdd P 49c4b16a457f45f28bef76b6a2b42faa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49c4b16a457f45f28bef76b6a2b42faa, 5abd0c1ef2ba4a019b905aa559b26c93; no voters: 951cdf4e3e67487da6c4eab75ed97eb4
I20260808 07:01:23.460968 17669 raft_consensus.cc:2781] T 3dce9bd5194b4c3ea05360f1512e3cdd P 49c4b16a457f45f28bef76b6a2b42faa [term 2 FOLLOWER]: Leader election won for term 2
I20260808 07:01:23.461119 17669 raft_consensus.cc:686] T 3dce9bd5194b4c3ea05360f1512e3cdd P 49c4b16a457f45f28bef76b6a2b42faa [term 2 LEADER]: Becoming Leader. State: Replica: 49c4b16a457f45f28bef76b6a2b42faa, State: Running, Role: LEADER
I20260808 07:01:23.461292 17669 consensus_queue.cc:237] T 3dce9bd5194b4c3ea05360f1512e3cdd P 49c4b16a457f45f28bef76b6a2b42faa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c4b16a457f45f28bef76b6a2b42faa" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 37955 } } peers { permanent_uuid: "5abd0c1ef2ba4a019b905aa559b26c93" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 33161 } } peers { permanent_uuid: "951cdf4e3e67487da6c4eab75ed97eb4" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 33901 } }
I20260808 07:01:23.462080 17102 catalog_manager.cc:5495] T 3dce9bd5194b4c3ea05360f1512e3cdd P 49c4b16a457f45f28bef76b6a2b42faa reported cstate change: term changed from 1 to 2, leader changed from 951cdf4e3e67487da6c4eab75ed97eb4 (127.15.250.130) to 49c4b16a457f45f28bef76b6a2b42faa (127.15.250.129). New cstate: current_term: 2 leader_uuid: "49c4b16a457f45f28bef76b6a2b42faa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c4b16a457f45f28bef76b6a2b42faa" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 37955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5abd0c1ef2ba4a019b905aa559b26c93" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 33161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "951cdf4e3e67487da6c4eab75ed97eb4" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 33901 } health_report { overall_health: UNKNOWN } } }
I20260808 07:01:23.465224 17539 raft_consensus.cc:1261] T 3dce9bd5194b4c3ea05360f1512e3cdd P 5abd0c1ef2ba4a019b905aa559b26c93 [term 2 FOLLOWER]: Refusing update from remote peer 49c4b16a457f45f28bef76b6a2b42faa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20260808 07:01:23.465411 17193 consensus_peers.cc:487] T 3dce9bd5194b4c3ea05360f1512e3cdd P 49c4b16a457f45f28bef76b6a2b42faa -> Peer 951cdf4e3e67487da6c4eab75ed97eb4 (127.15.250.130:33901): Couldn't send request to peer 951cdf4e3e67487da6c4eab75ed97eb4. Status: Network error: Client connection negotiation failed: client connection to 127.15.250.130:33901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 07:01:23.465468 17670 consensus_queue.cc:1035] T 3dce9bd5194b4c3ea05360f1512e3cdd P 49c4b16a457f45f28bef76b6a2b42faa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5abd0c1ef2ba4a019b905aa559b26c93" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 33161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 07:01:23.475682 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.130:33901
--local_ip_for_outbound_sockets=127.15.250.130
--tserver_master_addrs=127.15.250.190:44153
--webserver_port=35907
--webserver_interface=127.15.250.130
--builtin_ntp_servers=127.15.250.148:42217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
W20260808 07:01:23.562929 17679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:23.563171 17679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:23.563218 17679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:23.564507 17679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:23.564579 17679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.130
I20260808 07:01:23.566067 17679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:42217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.130:33901
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.250.130
--webserver_port=35907
--tserver_master_addrs=127.15.250.190:44153
--never_fsync=true
--heap_profile_path=/tmp/kudu.17679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.130
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:01:23.566342 17679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:23.566689 17679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:23.571753 17684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:23.571866 17679 server_base.cc:1034] running on GCE node
W20260808 07:01:23.571802 17685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:23.572005 17687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:23.572296 17679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:23.572588 17679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:23.573755 17679 hybrid_clock.cc:648] HybridClock initialized: now 1786172483573734 us; error 42 us; skew 500 ppm
I20260808 07:01:23.575073 17679 webserver.cc:460] Webserver started at http://127.15.250.130:35907/ using document root <none> and password file <none>
I20260808 07:01:23.575331 17679 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:23.575392 17679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:23.576902 17679 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 07:01:23.577601 17693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:23.577801 17679 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:01:23.577912 17679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "951cdf4e3e67487da6c4eab75ed97eb4"
format_stamp: "Formatted at 2026-08-08 07:01:22 on dist-test-slave-0b58"
I20260808 07:01:23.578253 17679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:23.607846 17679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:23.608208 17679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:23.608402 17679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:23.608659 17679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:01:23.609148 17700 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 07:01:23.610509 17679 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 07:01:23.610579 17679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260808 07:01:23.610630 17679 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 07:01:23.611338 17679 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 07:01:23.611398 17679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260808 07:01:23.611475 17700 tablet_bootstrap.cc:492] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4: Bootstrap starting.
I20260808 07:01:23.618558 17679 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.130:33901
I20260808 07:01:23.618716 17819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.130:33901 every 8 connection(s)
I20260808 07:01:23.618854 17700 log.cc:824] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:23.619050 17679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260808 07:01:23.622900 17700 tablet_bootstrap.cc:492] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=500 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 07:01:23.623333 17700 tablet_bootstrap.cc:492] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4: Bootstrap complete.
I20260808 07:01:23.623929 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 17679
I20260808 07:01:23.624537 17700 ts_tablet_manager.cc:1397] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.003s
I20260808 07:01:23.626243 17700 raft_consensus.cc:348] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c4b16a457f45f28bef76b6a2b42faa" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 37955 } } peers { permanent_uuid: "5abd0c1ef2ba4a019b905aa559b26c93" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 33161 } } peers { permanent_uuid: "951cdf4e3e67487da6c4eab75ed97eb4" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 33901 } }
I20260808 07:01:23.626566 17700 raft_consensus.cc:729] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951cdf4e3e67487da6c4eab75ed97eb4, State: Initialized, Role: FOLLOWER
I20260808 07:01:23.626716 17700 consensus_queue.cc:260] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c4b16a457f45f28bef76b6a2b42faa" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 37955 } } peers { permanent_uuid: "5abd0c1ef2ba4a019b905aa559b26c93" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 33161 } } peers { permanent_uuid: "951cdf4e3e67487da6c4eab75ed97eb4" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 33901 } }
I20260808 07:01:23.627084 17700 ts_tablet_manager.cc:1428] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260808 07:01:23.627125 17822 heartbeater.cc:347] Connected to a master server at 127.15.250.190:44153
I20260808 07:01:23.627202 17822 heartbeater.cc:464] Registering TS with master...
I20260808 07:01:23.627388 17822 heartbeater.cc:511] Master 127.15.250.190:44153 requested a full tablet report, sending...
I20260808 07:01:23.628043 17102 ts_manager.cc:194] Re-registered known tserver with Master: 951cdf4e3e67487da6c4eab75ed97eb4 (127.15.250.130:33901)
I20260808 07:01:23.628589 17102 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.130:59369
I20260808 07:01:23.629343 17822 heartbeater.cc:503] Master 127.15.250.190:44153 was elected leader, sending a full tablet report...
I20260808 07:01:23.652856 17239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260808 07:01:23.653330 17519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260808 07:01:23.656644 17747 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e0d1eb4f77554f4c8b3cb4c586625a7a | 127.15.250.190:44153 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.250.148:42217 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
49c4b16a457f45f28bef76b6a2b42faa | 127.15.250.129:37955 | HEALTHY | <none> | 1 | 0
5abd0c1ef2ba4a019b905aa559b26c93 | 127.15.250.131:33161 | HEALTHY | <none> | 0 | 0
951cdf4e3e67487da6c4eab75ed97eb4 | 127.15.250.130:33901 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.15.250.129 | experimental | 127.15.250.129:37955
local_ip_for_outbound_sockets | 127.15.250.130 | experimental | 127.15.250.130:33901
local_ip_for_outbound_sockets | 127.15.250.131 | experimental | 127.15.250.131:33161
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.15.250.129:37955
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.15.250.130:33901
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.15.250.131:33161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.250.148:42217 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 3dce9bd5194b4c3ea05360f1512e3cdd of table 'TestTable' is conflicted: 1 replicas' active configs disagree with the leader master's
49c4b16a457f45f28bef76b6a2b42faa (127.15.250.129:37955): RUNNING [LEADER]
5abd0c1ef2ba4a019b905aa559b26c93 (127.15.250.131:33161): RUNNING
951cdf4e3e67487da6c4eab75ed97eb4 (127.15.250.130:33901): RUNNING
All reported replicas are:
A = 49c4b16a457f45f28bef76b6a2b42faa
B = 5abd0c1ef2ba4a019b905aa559b26c93
C = 951cdf4e3e67487da6c4eab75ed97eb4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | A* B C | 2 | -1 | Yes
C | A B C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+--------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260808 07:01:23.663491 16362 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260808 07:01:23.840457 17239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260808 07:01:23.840731 17519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260808 07:01:23.840914 17747 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e0d1eb4f77554f4c8b3cb4c586625a7a | 127.15.250.190:44153 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.250.148:42217 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
49c4b16a457f45f28bef76b6a2b42faa | 127.15.250.129:37955 | HEALTHY | <none> | 1 | 0
5abd0c1ef2ba4a019b905aa559b26c93 | 127.15.250.131:33161 | HEALTHY | <none> | 0 | 0
951cdf4e3e67487da6c4eab75ed97eb4 | 127.15.250.130:33901 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.15.250.129 | experimental | 127.15.250.129:37955
local_ip_for_outbound_sockets | 127.15.250.130 | experimental | 127.15.250.130:33901
local_ip_for_outbound_sockets | 127.15.250.131 | experimental | 127.15.250.131:33161
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.15.250.129:37955
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.15.250.130:33901
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.15.250.131:33161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.250.148:42217 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 3dce9bd5194b4c3ea05360f1512e3cdd of table 'TestTable' is conflicted: 1 replicas' active configs disagree with the leader master's
49c4b16a457f45f28bef76b6a2b42faa (127.15.250.129:37955): RUNNING [LEADER]
5abd0c1ef2ba4a019b905aa559b26c93 (127.15.250.131:33161): RUNNING
951cdf4e3e67487da6c4eab75ed97eb4 (127.15.250.130:33901): RUNNING
All reported replicas are:
A = 49c4b16a457f45f28bef76b6a2b42faa
B = 5abd0c1ef2ba4a019b905aa559b26c93
C = 951cdf4e3e67487da6c4eab75ed97eb4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | A* B C | 2 | -1 | Yes
C | A B C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+--------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260808 07:01:23.848592 16362 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260808 07:01:24.022097 17771 raft_consensus.cc:3037] T 3dce9bd5194b4c3ea05360f1512e3cdd P 951cdf4e3e67487da6c4eab75ed97eb4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 07:01:24.025519 17880 mvcc.cc:205] Tried to move back new op lower bound from 7316162492311117824 to 7316162492258738176. Current Snapshot: MvccSnapshot[applied={T|T < 7316162492287107072}]
I20260808 07:01:24.101789 17239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260808 07:01:24.102025 17519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260808 07:01:24.102222 17747 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e0d1eb4f77554f4c8b3cb4c586625a7a | 127.15.250.190:44153 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.250.148:42217 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
49c4b16a457f45f28bef76b6a2b42faa | 127.15.250.129:37955 | HEALTHY | <none> | 1 | 0
5abd0c1ef2ba4a019b905aa559b26c93 | 127.15.250.131:33161 | HEALTHY | <none> | 0 | 0
951cdf4e3e67487da6c4eab75ed97eb4 | 127.15.250.130:33901 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.15.250.129 | experimental | 127.15.250.129:37955
local_ip_for_outbound_sockets | 127.15.250.130 | experimental | 127.15.250.130:33901
local_ip_for_outbound_sockets | 127.15.250.131 | experimental | 127.15.250.131:33161
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.15.250.129:37955
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.15.250.130:33901
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.15.250.131:33161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.250.148:42217 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260808 07:01:24.109939 16362 log_verifier.cc:126] Checking tablet 3dce9bd5194b4c3ea05360f1512e3cdd
I20260808 07:01:24.122129 16362 log_verifier.cc:177] Verified matching terms for 12 ops in tablet 3dce9bd5194b4c3ea05360f1512e3cdd
W20260808 07:01:24.649680 16362 scanner-internal.cc:458] Time spent opening tablet: real 0.522s user 0.000s sys 0.000s
I20260808 07:01:24.650483 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 17176
I20260808 07:01:24.656406 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 17679
I20260808 07:01:24.663281 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 17452
I20260808 07:01:24.669456 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 17078
2026-08-08T07:01:24Z chronyd exiting
[ OK ] RaftConsensusElectionITest.RunLeaderElection (3026 ms)
[ RUN ] RaftConsensusElectionITest.ChurnyElections
I20260808 07:01:24.692459 16362 test_util.cc:268] Using random seed: -1334498842
I20260808 07:01:24.693004 16362 ts_itest-base.cc:112] Starting cluster with:
I20260808 07:01:24.693051 16362 ts_itest-base.cc:113] --------------
I20260808 07:01:24.693063 16362 ts_itest-base.cc:114] 3 tablet servers
I20260808 07:01:24.693073 16362 ts_itest-base.cc:115] 3 replicas per TS
I20260808 07:01:24.693084 16362 ts_itest-base.cc:116] --------------
2026-08-08T07:01:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:01:24Z Disabled control of system clock
I20260808 07:01:24.702826 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.250.190:44265
--webserver_interface=127.15.250.190
--webserver_port=0
--builtin_ntp_servers=127.15.250.148:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.250.190:44265 with env {}
W20260808 07:01:24.786281 17927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:24.786507 17927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:24.786527 17927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:24.787667 17927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:01:24.787705 17927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:24.787725 17927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:01:24.787739 17927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:01:24.789186 17927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.250.190:44265
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.250.190:44265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.250.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.17927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:01:24.789409 17927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:24.789672 17927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:24.794953 17933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:24.794957 17932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:24.794957 17935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:24.795311 17927 server_base.cc:1034] running on GCE node
I20260808 07:01:24.795548 17927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:24.795818 17927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:24.796959 17927 hybrid_clock.cc:648] HybridClock initialized: now 1786172484796936 us; error 42 us; skew 500 ppm
I20260808 07:01:24.798286 17927 webserver.cc:460] Webserver started at http://127.15.250.190:41641/ using document root <none> and password file <none>
I20260808 07:01:24.798487 17927 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:24.798529 17927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:24.798621 17927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:24.799569 17927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "298377829a064c73bfa7cba5ab5bd72c"
format_stamp: "Formatted at 2026-08-08 07:01:24 on dist-test-slave-0b58"
I20260808 07:01:24.799898 17927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "298377829a064c73bfa7cba5ab5bd72c"
format_stamp: "Formatted at 2026-08-08 07:01:24 on dist-test-slave-0b58"
I20260808 07:01:24.801256 17927 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 07:01:24.801987 17942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:24.802183 17927 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:01:24.802299 17927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
uuid: "298377829a064c73bfa7cba5ab5bd72c"
format_stamp: "Formatted at 2026-08-08 07:01:24 on dist-test-slave-0b58"
I20260808 07:01:24.802378 17927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:24.816627 17927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:24.816947 17927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:24.817098 17927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:24.821004 17927 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.190:44265
I20260808 07:01:24.821043 17996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.190:44265 every 8 connection(s)
I20260808 07:01:24.821388 17927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260808 07:01:24.822079 17997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:24.824884 17997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 298377829a064c73bfa7cba5ab5bd72c: Bootstrap starting.
I20260808 07:01:24.825527 17997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 298377829a064c73bfa7cba5ab5bd72c: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:24.825845 17997 log.cc:824] T 00000000000000000000000000000000 P 298377829a064c73bfa7cba5ab5bd72c: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:24.826632 17997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 298377829a064c73bfa7cba5ab5bd72c: No bootstrap required, opened a new log
I20260808 07:01:24.827628 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 17927
I20260808 07:01:24.827764 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260808 07:01:24.828436 17997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 298377829a064c73bfa7cba5ab5bd72c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298377829a064c73bfa7cba5ab5bd72c" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 44265 } }
I20260808 07:01:24.828699 17997 raft_consensus.cc:374] T 00000000000000000000000000000000 P 298377829a064c73bfa7cba5ab5bd72c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:01:24.828797 17997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 298377829a064c73bfa7cba5ab5bd72c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 298377829a064c73bfa7cba5ab5bd72c, State: Initialized, Role: FOLLOWER
I20260808 07:01:24.828955 17997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 298377829a064c73bfa7cba5ab5bd72c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298377829a064c73bfa7cba5ab5bd72c" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 44265 } }
I20260808 07:01:24.829056 17997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 298377829a064c73bfa7cba5ab5bd72c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:01:24.829120 17997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 298377829a064c73bfa7cba5ab5bd72c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:01:24.829198 17997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 298377829a064c73bfa7cba5ab5bd72c [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:01:24.829802 17997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 298377829a064c73bfa7cba5ab5bd72c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298377829a064c73bfa7cba5ab5bd72c" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 44265 } }
I20260808 07:01:24.829924 17997 leader_election.cc:302] T 00000000000000000000000000000000 P 298377829a064c73bfa7cba5ab5bd72c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 298377829a064c73bfa7cba5ab5bd72c; no voters:
I20260808 07:01:24.830102 17997 leader_election.cc:288] T 00000000000000000000000000000000 P 298377829a064c73bfa7cba5ab5bd72c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:01:24.830140 18004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 298377829a064c73bfa7cba5ab5bd72c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:01:24.830260 18004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 298377829a064c73bfa7cba5ab5bd72c [term 1 LEADER]: Becoming Leader. State: Replica: 298377829a064c73bfa7cba5ab5bd72c, State: Running, Role: LEADER
I20260808 07:01:24.830372 17997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 298377829a064c73bfa7cba5ab5bd72c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:01:24.830399 18004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 298377829a064c73bfa7cba5ab5bd72c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298377829a064c73bfa7cba5ab5bd72c" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 44265 } }
I20260808 07:01:24.830721 18004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 298377829a064c73bfa7cba5ab5bd72c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 298377829a064c73bfa7cba5ab5bd72c. Latest consensus state: current_term: 1 leader_uuid: "298377829a064c73bfa7cba5ab5bd72c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298377829a064c73bfa7cba5ab5bd72c" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 44265 } } }
I20260808 07:01:24.830813 18004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 298377829a064c73bfa7cba5ab5bd72c [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:24.830735 18006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 298377829a064c73bfa7cba5ab5bd72c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "298377829a064c73bfa7cba5ab5bd72c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298377829a064c73bfa7cba5ab5bd72c" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 44265 } } }
I20260808 07:01:24.830924 18006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 298377829a064c73bfa7cba5ab5bd72c [sys.catalog]: This master's current role is: LEADER
W20260808 07:01:24.832664 18021 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 298377829a064c73bfa7cba5ab5bd72c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 07:01:24.832790 18021 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 07:01:24.832863 18010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:01:24.833045 18010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:01:24.834779 18010 catalog_manager.cc:1347] Generated new cluster ID: 1027a64dc22e40a5a8c0e08bd7946afd
I20260808 07:01:24.834846 18010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:01:24.845762 18010 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:01:24.846383 18010 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:01:24.850056 18010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 298377829a064c73bfa7cba5ab5bd72c: Generated new TSK 0
I20260808 07:01:24.850260 18010 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:01:24.851353 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.129:0
--local_ip_for_outbound_sockets=127.15.250.129
--webserver_interface=127.15.250.129
--webserver_port=0
--tserver_master_addrs=127.15.250.190:44265
--builtin_ntp_servers=127.15.250.148:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
W20260808 07:01:24.936192 18024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:24.936482 18024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:24.936506 18024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:24.937748 18024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:24.937808 18024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.129
I20260808 07:01:24.939222 18024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.250.129
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.15.250.190:44265
--never_fsync=true
--heap_profile_path=/tmp/kudu.18024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.129
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:01:24.939468 18024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:24.939805 18024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:24.945421 18029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:24.945422 18030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:24.945459 18032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:24.945907 18024 server_base.cc:1034] running on GCE node
I20260808 07:01:24.946168 18024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:24.946432 18024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:24.947599 18024 hybrid_clock.cc:648] HybridClock initialized: now 1786172484947582 us; error 32 us; skew 500 ppm
I20260808 07:01:24.949194 18024 webserver.cc:460] Webserver started at http://127.15.250.129:40471/ using document root <none> and password file <none>
I20260808 07:01:24.949509 18024 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:24.949570 18024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:24.949712 18024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:24.951102 18024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "910bd273924d4cb0adc0f7b65e454f19"
format_stamp: "Formatted at 2026-08-08 07:01:24 on dist-test-slave-0b58"
I20260808 07:01:24.951530 18024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "910bd273924d4cb0adc0f7b65e454f19"
format_stamp: "Formatted at 2026-08-08 07:01:24 on dist-test-slave-0b58"
I20260808 07:01:24.953325 18024 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260808 07:01:24.954424 18038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:24.954663 18024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:01:24.954770 18024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "910bd273924d4cb0adc0f7b65e454f19"
format_stamp: "Formatted at 2026-08-08 07:01:24 on dist-test-slave-0b58"
I20260808 07:01:24.954844 18024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:24.968019 18024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:24.968394 18024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:24.968509 18024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:24.968731 18024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:01:24.969079 18024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:01:24.969111 18024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:24.969172 18024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:01:24.969200 18024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:24.975070 18024 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.129:45605
I20260808 07:01:24.975121 18160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.129:45605 every 8 connection(s)
I20260808 07:01:24.975458 18024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260808 07:01:24.977033 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 18024
I20260808 07:01:24.977152 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260808 07:01:24.978361 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.130:0
--local_ip_for_outbound_sockets=127.15.250.130
--webserver_interface=127.15.250.130
--webserver_port=0
--tserver_master_addrs=127.15.250.190:44265
--builtin_ntp_servers=127.15.250.148:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
I20260808 07:01:24.982044 18161 heartbeater.cc:347] Connected to a master server at 127.15.250.190:44265
I20260808 07:01:24.982206 18161 heartbeater.cc:464] Registering TS with master...
I20260808 07:01:24.982429 18161 heartbeater.cc:511] Master 127.15.250.190:44265 requested a full tablet report, sending...
I20260808 07:01:24.982975 17959 ts_manager.cc:194] Registered new tserver with Master: 910bd273924d4cb0adc0f7b65e454f19 (127.15.250.129:45605)
I20260808 07:01:24.983752 17959 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.129:47471
W20260808 07:01:25.064595 18164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:25.064839 18164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:25.064878 18164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:25.066318 18164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:25.066396 18164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.130
I20260808 07:01:25.067867 18164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.250.130
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.15.250.190:44265
--never_fsync=true
--heap_profile_path=/tmp/kudu.18164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.130
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:01:25.068070 18164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:25.068405 18164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:25.073321 18172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:25.073320 18169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:25.073414 18170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:25.073800 18164 server_base.cc:1034] running on GCE node
I20260808 07:01:25.074012 18164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:25.074206 18164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:25.075390 18164 hybrid_clock.cc:648] HybridClock initialized: now 1786172485075368 us; error 29 us; skew 500 ppm
I20260808 07:01:25.076874 18164 webserver.cc:460] Webserver started at http://127.15.250.130:35443/ using document root <none> and password file <none>
I20260808 07:01:25.077121 18164 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:25.077176 18164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:25.077338 18164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:25.078408 18164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "03f850c83dff43fdade5214b4dbee7a1"
format_stamp: "Formatted at 2026-08-08 07:01:25 on dist-test-slave-0b58"
I20260808 07:01:25.078790 18164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "03f850c83dff43fdade5214b4dbee7a1"
format_stamp: "Formatted at 2026-08-08 07:01:25 on dist-test-slave-0b58"
I20260808 07:01:25.080231 18164 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260808 07:01:25.081085 18178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:25.081300 18164 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:01:25.081414 18164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "03f850c83dff43fdade5214b4dbee7a1"
format_stamp: "Formatted at 2026-08-08 07:01:25 on dist-test-slave-0b58"
I20260808 07:01:25.081496 18164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:25.100723 18164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:25.101047 18164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:25.101219 18164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:25.101465 18164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:01:25.101818 18164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:01:25.101877 18164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:25.101912 18164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:01:25.101960 18164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:25.108110 18164 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.130:43273
I20260808 07:01:25.108157 18298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.130:43273 every 8 connection(s)
I20260808 07:01:25.108564 18164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260808 07:01:25.114207 18300 heartbeater.cc:347] Connected to a master server at 127.15.250.190:44265
I20260808 07:01:25.114348 18300 heartbeater.cc:464] Registering TS with master...
I20260808 07:01:25.114537 18300 heartbeater.cc:511] Master 127.15.250.190:44265 requested a full tablet report, sending...
I20260808 07:01:25.114966 17959 ts_manager.cc:194] Registered new tserver with Master: 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273)
I20260808 07:01:25.114993 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 18164
I20260808 07:01:25.115103 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260808 07:01:25.115396 17959 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.130:43807
I20260808 07:01:25.116207 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.131:0
--local_ip_for_outbound_sockets=127.15.250.131
--webserver_interface=127.15.250.131
--webserver_port=0
--tserver_master_addrs=127.15.250.190:44265
--builtin_ntp_servers=127.15.250.148:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
W20260808 07:01:25.200575 18304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:25.200788 18304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:25.200810 18304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:25.202047 18304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:25.202105 18304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.131
I20260808 07:01:25.203488 18304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.250.131
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.15.250.190:44265
--never_fsync=true
--heap_profile_path=/tmp/kudu.18304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.131
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:01:25.203681 18304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:25.203933 18304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:25.209024 18310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:25.209033 18312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:25.209033 18309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:25.209251 18304 server_base.cc:1034] running on GCE node
I20260808 07:01:25.209457 18304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:25.209671 18304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:25.210860 18304 hybrid_clock.cc:648] HybridClock initialized: now 1786172485210839 us; error 35 us; skew 500 ppm
I20260808 07:01:25.212203 18304 webserver.cc:460] Webserver started at http://127.15.250.131:42457/ using document root <none> and password file <none>
I20260808 07:01:25.212509 18304 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:25.212594 18304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:25.212751 18304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:25.213742 18304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "221e532665ab4c7f9585be8ed849b6a9"
format_stamp: "Formatted at 2026-08-08 07:01:25 on dist-test-slave-0b58"
I20260808 07:01:25.214110 18304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "221e532665ab4c7f9585be8ed849b6a9"
format_stamp: "Formatted at 2026-08-08 07:01:25 on dist-test-slave-0b58"
I20260808 07:01:25.215489 18304 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 07:01:25.216200 18318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:25.216394 18304 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:01:25.216492 18304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "221e532665ab4c7f9585be8ed849b6a9"
format_stamp: "Formatted at 2026-08-08 07:01:25 on dist-test-slave-0b58"
I20260808 07:01:25.216594 18304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:25.226847 18304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:25.227093 18304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:25.227193 18304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:25.227375 18304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:01:25.227648 18304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:01:25.227679 18304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:25.227699 18304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:01:25.227712 18304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:25.234108 18304 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.131:44581
I20260808 07:01:25.234164 18437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.131:44581 every 8 connection(s)
I20260808 07:01:25.234486 18304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260808 07:01:25.240304 18438 heartbeater.cc:347] Connected to a master server at 127.15.250.190:44265
I20260808 07:01:25.240402 18438 heartbeater.cc:464] Registering TS with master...
I20260808 07:01:25.240595 18438 heartbeater.cc:511] Master 127.15.250.190:44265 requested a full tablet report, sending...
I20260808 07:01:25.241012 17959 ts_manager.cc:194] Registered new tserver with Master: 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:25.241410 17959 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.131:44179
I20260808 07:01:25.241943 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 18304
I20260808 07:01:25.242025 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260808 07:01:25.243099 16362 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 07:01:25.244230 16362 test_util.cc:268] Using random seed: -1333947070
I20260808 07:01:25.251355 17959 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34906:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 07:01:25.251726 17959 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 07:01:25.259330 18225 tablet_service.cc:1467] Processing CreateTablet for tablet fcb881fce4ea471890e23b30bc4f2eb4 (DEFAULT_TABLE table=test-workload [id=07a3f1c7a680491790cdcb91d1675971]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:01:25.259353 18091 tablet_service.cc:1467] Processing CreateTablet for tablet fcb881fce4ea471890e23b30bc4f2eb4 (DEFAULT_TABLE table=test-workload [id=07a3f1c7a680491790cdcb91d1675971]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:01:25.259764 18225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcb881fce4ea471890e23b30bc4f2eb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:25.259768 18091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcb881fce4ea471890e23b30bc4f2eb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:25.262396 18453 tablet_bootstrap.cc:492] T fcb881fce4ea471890e23b30bc4f2eb4 P 03f850c83dff43fdade5214b4dbee7a1: Bootstrap starting.
I20260808 07:01:25.262956 18367 tablet_service.cc:1467] Processing CreateTablet for tablet fcb881fce4ea471890e23b30bc4f2eb4 (DEFAULT_TABLE table=test-workload [id=07a3f1c7a680491790cdcb91d1675971]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:01:25.263278 18453 tablet_bootstrap.cc:654] T fcb881fce4ea471890e23b30bc4f2eb4 P 03f850c83dff43fdade5214b4dbee7a1: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:25.263302 18367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcb881fce4ea471890e23b30bc4f2eb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:25.263610 18453 log.cc:824] T fcb881fce4ea471890e23b30bc4f2eb4 P 03f850c83dff43fdade5214b4dbee7a1: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:25.264006 18452 tablet_bootstrap.cc:492] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19: Bootstrap starting.
I20260808 07:01:25.264454 18453 tablet_bootstrap.cc:492] T fcb881fce4ea471890e23b30bc4f2eb4 P 03f850c83dff43fdade5214b4dbee7a1: No bootstrap required, opened a new log
I20260808 07:01:25.264546 18453 ts_tablet_manager.cc:1397] T fcb881fce4ea471890e23b30bc4f2eb4 P 03f850c83dff43fdade5214b4dbee7a1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:01:25.264755 18452 tablet_bootstrap.cc:654] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:25.265087 18452 log.cc:824] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:25.265878 18452 tablet_bootstrap.cc:492] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19: No bootstrap required, opened a new log
I20260808 07:01:25.265951 18452 ts_tablet_manager.cc:1397] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:01:25.266855 18456 tablet_bootstrap.cc:492] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9: Bootstrap starting.
I20260808 07:01:25.267168 18453 raft_consensus.cc:348] T fcb881fce4ea471890e23b30bc4f2eb4 P 03f850c83dff43fdade5214b4dbee7a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:25.267326 18453 raft_consensus.cc:374] T fcb881fce4ea471890e23b30bc4f2eb4 P 03f850c83dff43fdade5214b4dbee7a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:01:25.267367 18453 raft_consensus.cc:729] T fcb881fce4ea471890e23b30bc4f2eb4 P 03f850c83dff43fdade5214b4dbee7a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03f850c83dff43fdade5214b4dbee7a1, State: Initialized, Role: FOLLOWER
I20260808 07:01:25.267565 18456 tablet_bootstrap.cc:654] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:25.267570 18453 consensus_queue.cc:260] T fcb881fce4ea471890e23b30bc4f2eb4 P 03f850c83dff43fdade5214b4dbee7a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:25.267766 18457 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 03f850c83dff43fdade5214b4dbee7a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 07:01:25.267866 18453 ts_tablet_manager.cc:1428] T fcb881fce4ea471890e23b30bc4f2eb4 P 03f850c83dff43fdade5214b4dbee7a1: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 07:01:25.267902 18456 log.cc:824] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:25.267995 18300 heartbeater.cc:503] Master 127.15.250.190:44265 was elected leader, sending a full tablet report...
I20260808 07:01:25.267882 18457 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 03f850c83dff43fdade5214b4dbee7a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:25.268585 18456 tablet_bootstrap.cc:492] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9: No bootstrap required, opened a new log
I20260808 07:01:25.268689 18457 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 03f850c83dff43fdade5214b4dbee7a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 910bd273924d4cb0adc0f7b65e454f19 (127.15.250.129:45605), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:25.268929 18456 ts_tablet_manager.cc:1397] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 07:01:25.269500 18452 raft_consensus.cc:348] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:25.269670 18452 raft_consensus.cc:374] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:01:25.269933 18452 raft_consensus.cc:729] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 910bd273924d4cb0adc0f7b65e454f19, State: Initialized, Role: FOLLOWER
I20260808 07:01:25.270151 18452 consensus_queue.cc:260] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:25.270309 18461 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 07:01:25.270391 18461 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:25.270457 18452 ts_tablet_manager.cc:1428] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260808 07:01:25.270567 18161 heartbeater.cc:503] Master 127.15.250.190:44265 was elected leader, sending a full tablet report...
I20260808 07:01:25.271631 18456 raft_consensus.cc:348] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:25.271812 18456 raft_consensus.cc:374] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:01:25.271865 18456 raft_consensus.cc:729] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 221e532665ab4c7f9585be8ed849b6a9, State: Initialized, Role: FOLLOWER
I20260808 07:01:25.272053 18456 consensus_queue.cc:260] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:25.272151 18465 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 07:01:25.272198 18465 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:25.272503 18456 ts_tablet_manager.cc:1428] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260808 07:01:25.272614 18438 heartbeater.cc:503] Master 127.15.250.190:44265 was elected leader, sending a full tablet report...
I20260808 07:01:25.273368 18465 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 910bd273924d4cb0adc0f7b65e454f19 (127.15.250.129:45605), 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273)
I20260808 07:01:25.275188 18112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "03f850c83dff43fdade5214b4dbee7a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "910bd273924d4cb0adc0f7b65e454f19" is_pre_election: true
I20260808 07:01:25.275308 18112 raft_consensus.cc:2454] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03f850c83dff43fdade5214b4dbee7a1 in term 0.
W20260808 07:01:25.275517 18182 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.15.250.129:45605 timed out after 0.007s (SENT)
I20260808 07:01:25.276010 18461 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:25.276340 18112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "03f850c83dff43fdade5214b4dbee7a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "910bd273924d4cb0adc0f7b65e454f19" is_pre_election: true
I20260808 07:01:25.276417 18112 raft_consensus.cc:2454] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03f850c83dff43fdade5214b4dbee7a1 in term 0.
I20260808 07:01:25.276367 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:25.276481 18251 raft_consensus.cc:2454] T fcb881fce4ea471890e23b30bc4f2eb4 P 03f850c83dff43fdade5214b4dbee7a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910bd273924d4cb0adc0f7b65e454f19 in term 0.
I20260808 07:01:25.276612 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "03f850c83dff43fdade5214b4dbee7a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:25.276614 18182 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 03f850c83dff43fdade5214b4dbee7a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03f850c83dff43fdade5214b4dbee7a1, 910bd273924d4cb0adc0f7b65e454f19; no voters:
I20260808 07:01:25.276697 18389 raft_consensus.cc:2454] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03f850c83dff43fdade5214b4dbee7a1 in term 0.
I20260808 07:01:25.276767 18458 raft_consensus.cc:2781] T fcb881fce4ea471890e23b30bc4f2eb4 P 03f850c83dff43fdade5214b4dbee7a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 07:01:25.276824 18458 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 03f850c83dff43fdade5214b4dbee7a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 07:01:25.276856 18458 raft_consensus.cc:3037] T fcb881fce4ea471890e23b30bc4f2eb4 P 03f850c83dff43fdade5214b4dbee7a1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:01:25.277757 18458 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 03f850c83dff43fdade5214b4dbee7a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:25.277925 18458 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 03f850c83dff43fdade5214b4dbee7a1 [CANDIDATE]: Term 1 election: Requested vote from peers 910bd273924d4cb0adc0f7b65e454f19 (127.15.250.129:45605), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
W20260808 07:01:25.278074 18042 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.15.250.130:43273 timed out after 0.007s (SENT)
I20260808 07:01:25.278118 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "03f850c83dff43fdade5214b4dbee7a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9"
I20260808 07:01:25.278200 18389 raft_consensus.cc:3037] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:01:25.278215 18112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "910bd273924d4cb0adc0f7b65e454f19" is_pre_election: true
I20260808 07:01:25.278290 18112 raft_consensus.cc:2454] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 221e532665ab4c7f9585be8ed849b6a9 in term 0.
I20260808 07:01:25.278424 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:25.278532 18251 raft_consensus.cc:2379] T fcb881fce4ea471890e23b30bc4f2eb4 P 03f850c83dff43fdade5214b4dbee7a1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 221e532665ab4c7f9585be8ed849b6a9 in current term 1: Already voted for candidate 03f850c83dff43fdade5214b4dbee7a1 in this term.
W20260808 07:01:25.278741 18322 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.15.250.129:45605 timed out after 0.006s (SENT)
I20260808 07:01:25.279032 18389 raft_consensus.cc:2454] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03f850c83dff43fdade5214b4dbee7a1 in term 1.
I20260808 07:01:25.279004 18388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "03f850c83dff43fdade5214b4dbee7a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:25.279105 18388 raft_consensus.cc:2362] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 03f850c83dff43fdade5214b4dbee7a1 in term 1. Re-sending same reply.
I20260808 07:01:25.279101 18112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "03f850c83dff43fdade5214b4dbee7a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "910bd273924d4cb0adc0f7b65e454f19"
I20260808 07:01:25.279158 18112 raft_consensus.cc:3037] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:01:25.279166 18111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "910bd273924d4cb0adc0f7b65e454f19" is_pre_election: true
I20260808 07:01:25.279863 18112 raft_consensus.cc:2454] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03f850c83dff43fdade5214b4dbee7a1 in term 1.
I20260808 07:01:25.280048 18181 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 03f850c83dff43fdade5214b4dbee7a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9; no voters:
I20260808 07:01:25.280063 18322 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 221e532665ab4c7f9585be8ed849b6a9; no voters: 03f850c83dff43fdade5214b4dbee7a1, 910bd273924d4cb0adc0f7b65e454f19
I20260808 07:01:25.280184 18466 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 07:01:25.280226 18458 raft_consensus.cc:2781] T fcb881fce4ea471890e23b30bc4f2eb4 P 03f850c83dff43fdade5214b4dbee7a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:01:25.280318 18458 raft_consensus.cc:686] T fcb881fce4ea471890e23b30bc4f2eb4 P 03f850c83dff43fdade5214b4dbee7a1 [term 1 LEADER]: Becoming Leader. State: Replica: 03f850c83dff43fdade5214b4dbee7a1, State: Running, Role: LEADER
I20260808 07:01:25.280400 18458 consensus_queue.cc:237] T fcb881fce4ea471890e23b30bc4f2eb4 P 03f850c83dff43fdade5214b4dbee7a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:25.281369 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:25.281268 17959 catalog_manager.cc:5495] T fcb881fce4ea471890e23b30bc4f2eb4 P 03f850c83dff43fdade5214b4dbee7a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130). New cstate: current_term: 1 leader_uuid: "03f850c83dff43fdade5214b4dbee7a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } health_report { overall_health: UNKNOWN } } }
I20260808 07:01:25.282529 18457 consensus_queue.cc:1035] T fcb881fce4ea471890e23b30bc4f2eb4 P 03f850c83dff43fdade5214b4dbee7a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:01:25.282683 18388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:25.282778 18388 raft_consensus.cc:2379] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910bd273924d4cb0adc0f7b65e454f19 in current term 1: Already voted for candidate 03f850c83dff43fdade5214b4dbee7a1 in this term.
I20260808 07:01:25.283190 18458 consensus_queue.cc:1035] T fcb881fce4ea471890e23b30bc4f2eb4 P 03f850c83dff43fdade5214b4dbee7a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:01:25.283552 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:25.283761 18041 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:25.284135 18461 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 07:01:25.290740 18482 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260808 07:01:25.290791 18482 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20260808 07:01:25.856689 18388 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260808 07:01:25.856710 18112 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260808 07:01:25.856805 18388 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20260808 07:01:25.856819 18112 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260808 07:01:26.546720 18488 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:26.546833 18488 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:26.547320 18488 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:26.547533 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:26.547513 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:26.547932 18041 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:26.548179 18494 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=1, attempt_no=0}) took 1258 ms. Trace:
I20260808 07:01:26.548276 18494 rpcz_store.cc:276] 0808 07:01:25.289893 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:25.289916 (+ 23us) service_pool.cc:227] Handling call
0808 07:01:26.548171 (+1258255us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:25.290532 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:25.290550 (+ 18us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:25.290551 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:01:25.290551 (+ 0us) tablet.cc:660] Decoding operations
0808 07:01:25.290628 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:25.290646 (+ 18us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:25.290652 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:25.290706 (+ 54us) tablet.cc:699] Row locks acquired
0808 07:01:25.290706 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:01:25.290718 (+ 12us) write_op.cc:270] Start()
0808 07:01:25.855926 (+565208us) write_op.cc:276] Timestamp: P: 1786172485290693 usec, L: 0
0808 07:01:25.855930 (+ 4us) op_driver.cc:331] REPLICATION: starting
0808 07:01:25.856142 (+ 212us) log.cc:842] Serialized 345 byte log entry
0808 07:01:26.547604 (+691462us) op_driver.cc:444] REPLICATION: finished
0808 07:01:26.547752 (+ 148us) write_op.cc:301] APPLY: starting
0808 07:01:26.547771 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:26.547794 (+ 23us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:26.547797 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:26.547859 (+ 62us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:26.547897 (+ 38us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:26.547898 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:01:26.547984 (+ 86us) log.cc:842] Serialized 22 byte log entry
0808 07:01:26.548051 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:26.548077 (+ 26us) write_op.cc:454] Released schema lock
0808 07:01:26.548078 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":57088,"child_traces":[["op",{"apply.queue_time_us":123,"num_ops":1,"prepare.queue_time_us":40,"prepare.run_cpu_time_us":621,"prepare.run_wall_time_us":565692,"raft.queue_time_us":86,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":153,"replication_time_us":691651,"spinlock_wait_cycles":1280,"thread_start_us":78,"threads_started":1}]]}
I20260808 07:01:26.548745 18208 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60832 (request call id 2) took 1112 ms. Trace:
I20260808 07:01:26.548826 18208 rpcz_store.cc:276] 0808 07:01:25.436152 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:25.436182 (+ 30us) service_pool.cc:227] Handling call
0808 07:01:25.436378 (+ 196us) tablet_service.cc:2780] Created scanner 5a766b17a91d4597a9114b4295103f03 for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is acee7b44dd2a4886ada89174aec4335f
0808 07:01:25.436956 (+ 578us) tablet_service.cc:2920] Creating iterator
0808 07:01:25.436976 (+ 20us) tablet_service.cc:3297] Waiting safe time to advance
0808 07:01:25.856028 (+419052us) tablet_service.cc:3304] Waiting for operations to commit
0808 07:01:26.548087 (+692059us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1111103 microseconds
0808 07:01:26.548135 (+ 48us) tablet_service.cc:2945] Iterator created
0808 07:01:26.548381 (+ 246us) tablet_service.cc:2967] Iterator init: OK
0808 07:01:26.548395 (+ 14us) tablet_service.cc:3010] has_more: true
0808 07:01:26.548402 (+ 7us) tablet_service.cc:3027] Continuing scan request
0808 07:01:26.548430 (+ 28us) tablet_service.cc:3091] Found scanner 5a766b17a91d4597a9114b4295103f03 for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is acee7b44dd2a4886ada89174aec4335f
0808 07:01:26.548738 (+ 308us) inbound_call.cc:173] Queueing success response
Metrics: {"compiler_manager_pool.queue_time_us":407,"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":17536,"thread_start_us":87,"threads_started":1}
I20260808 07:01:26.548743 18207 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60832 (request call id 3) took 1112 ms. Trace:
I20260808 07:01:26.548985 18207 rpcz_store.cc:276] 0808 07:01:25.436285 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:25.436324 (+ 39us) service_pool.cc:227] Handling call
0808 07:01:25.436378 (+ 54us) tablet_service.cc:2780] Created scanner 5f9a7010af524016af4853645730a190 for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is cac3d9fbc0024ebb992001f0d79ee4de
0808 07:01:25.436957 (+ 579us) tablet_service.cc:2920] Creating iterator
0808 07:01:25.436976 (+ 19us) tablet_service.cc:3297] Waiting safe time to advance
0808 07:01:25.856015 (+419039us) tablet_service.cc:3304] Waiting for operations to commit
0808 07:01:26.548113 (+692098us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1111134 microseconds
0808 07:01:26.548132 (+ 19us) tablet_service.cc:2945] Iterator created
0808 07:01:26.548376 (+ 244us) tablet_service.cc:2967] Iterator init: OK
0808 07:01:26.548393 (+ 17us) tablet_service.cc:3010] has_more: true
0808 07:01:26.548403 (+ 10us) tablet_service.cc:3027] Continuing scan request
0808 07:01:26.548430 (+ 27us) tablet_service.cc:3091] Found scanner 5f9a7010af524016af4853645730a190 for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is cac3d9fbc0024ebb992001f0d79ee4de
0808 07:01:26.548737 (+ 307us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":48256}
W20260808 07:01:26.549466 18478 scanner-internal.cc:458] Time spent opening tablet: real 1.114s user 0.000s sys 0.000s
W20260808 07:01:26.549466 18477 scanner-internal.cc:458] Time spent opening tablet: real 1.114s user 0.000s sys 0.000s
I20260808 07:01:26.687366 18488 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:26.688164 18494 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=0, attempt_no=0}) took 1398 ms. Trace:
I20260808 07:01:26.688251 18494 rpcz_store.cc:276] 0808 07:01:25.289860 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:25.289901 (+ 41us) service_pool.cc:227] Handling call
0808 07:01:26.688152 (+1398251us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:25.856214 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:25.856230 (+ 16us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:25.856231 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:01:25.856231 (+ 0us) tablet.cc:660] Decoding operations
0808 07:01:25.856244 (+ 13us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:25.856276 (+ 32us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:25.856278 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:25.856289 (+ 11us) tablet.cc:699] Row locks acquired
0808 07:01:25.856290 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:01:25.856295 (+ 5us) write_op.cc:270] Start()
0808 07:01:26.437418 (+581123us) write_op.cc:276] Timestamp: P: 1786172485856241 usec, L: 0
0808 07:01:26.437421 (+ 3us) op_driver.cc:331] REPLICATION: starting
0808 07:01:26.437531 (+ 110us) log.cc:842] Serialized 345 byte log entry
0808 07:01:26.687904 (+250373us) op_driver.cc:444] REPLICATION: finished
0808 07:01:26.688029 (+ 125us) write_op.cc:301] APPLY: starting
0808 07:01:26.688033 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:26.688037 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:26.688040 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:26.688071 (+ 31us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:26.688078 (+ 7us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:26.688078 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:01:26.688101 (+ 23us) log.cc:842] Serialized 22 byte log entry
0808 07:01:26.688124 (+ 23us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:26.688127 (+ 3us) write_op.cc:454] Released schema lock
0808 07:01:26.688127 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":384,"child_traces":[["op",{"apply.queue_time_us":98,"mutex_wait_us":41,"num_ops":1,"prepare.queue_time_us":565744,"prepare.run_cpu_time_us":267,"prepare.run_wall_time_us":581345,"replication_time_us":250477}]]}
I20260808 07:01:26.730824 18487 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:26.730932 18487 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:26.731362 18487 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 910bd273924d4cb0adc0f7b65e454f19 (127.15.250.129:45605), 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273)
I20260808 07:01:26.731580 18111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "910bd273924d4cb0adc0f7b65e454f19" is_pre_election: true
I20260808 07:01:26.731642 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:26.731918 18322 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 221e532665ab4c7f9585be8ed849b6a9; no voters: 03f850c83dff43fdade5214b4dbee7a1, 910bd273924d4cb0adc0f7b65e454f19
I20260808 07:01:27.609629 18488 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:27.609753 18488 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:27.610000 18488 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:27.610224 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:27.610208 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:27.610577 18042 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:27.610678 18488 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:27.610983 18503 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=2, attempt_no=0}) took 1061 ms. Trace:
W20260808 07:01:27.611033 18477 scanner-internal.cc:458] Time spent opening tablet: real 0.911s user 0.000s sys 0.000s
I20260808 07:01:27.611059 18503 rpcz_store.cc:276] 0808 07:01:26.549403 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:26.549427 (+ 24us) service_pool.cc:227] Handling call
0808 07:01:27.610973 (+1061546us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:26.549557 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:26.549567 (+ 10us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:26.549568 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:01:26.549568 (+ 0us) tablet.cc:660] Decoding operations
0808 07:01:26.549577 (+ 9us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:26.549578 (+ 1us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:26.549579 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:26.549586 (+ 7us) tablet.cc:699] Row locks acquired
0808 07:01:26.549586 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:01:26.549590 (+ 4us) write_op.cc:270] Start()
0808 07:01:26.716697 (+167107us) write_op.cc:276] Timestamp: P: 1786172486549538 usec, L: 0
0808 07:01:26.716699 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:01:26.716788 (+ 89us) log.cc:842] Serialized 354 byte log entry
0808 07:01:27.610319 (+893531us) op_driver.cc:444] REPLICATION: finished
0808 07:01:27.610396 (+ 77us) write_op.cc:301] APPLY: starting
0808 07:01:27.610399 (+ 3us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:27.610404 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:27.610407 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:27.610419 (+ 12us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:27.610424 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:27.610425 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:01:27.610442 (+ 17us) log.cc:842] Serialized 22 byte log entry
0808 07:01:27.610948 (+ 506us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:27.610951 (+ 3us) write_op.cc:454] Released schema lock
0808 07:01:27.610952 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":65,"num_ops":1,"prepare.queue_time_us":23,"prepare.run_cpu_time_us":227,"prepare.run_wall_time_us":167305,"raft.queue_time_us":43,"raft.run_cpu_time_us":48,"raft.run_wall_time_us":49,"replication_time_us":893615,"thread_start_us":51,"threads_started":1}]]}
W20260808 07:01:27.611033 18478 scanner-internal.cc:458] Time spent opening tablet: real 0.911s user 0.000s sys 0.000s
I20260808 07:01:27.799616 18487 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:27.799734 18388 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.15.250.130:38799 (request call id 393) took 1068 ms. Trace:
I20260808 07:01:27.799847 18388 rpcz_store.cc:276] 0808 07:01:26.731524 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:26.731582 (+ 58us) service_pool.cc:227] Handling call
0808 07:01:26.731627 (+ 45us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 07:01:26.731646 (+ 19us) raft_consensus.cc:1520] Early marking committed up to index 2
0808 07:01:26.731755 (+ 109us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 07:01:26.731801 (+ 46us) write_op.cc:270] Start()
0808 07:01:26.828942 (+ 97141us) write_op.cc:276] Timestamp: P: 1786172485856241 usec, L: 0
0808 07:01:26.829024 (+ 82us) write_op.cc:270] Start()
0808 07:01:27.799157 (+970133us) write_op.cc:276] Timestamp: P: 1786172486549538 usec, L: 0
0808 07:01:27.799365 (+ 208us) log.cc:842] Serialized 699 byte log entry
0808 07:01:27.799437 (+ 72us) raft_consensus.cc:1631] Marking committed up to 3
0808 07:01:27.799519 (+ 82us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 07:01:27.799535 (+ 16us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 07:01:27.799706 (+ 171us) raft_consensus.cc:1671] finished
0808 07:01:27.799706 (+ 0us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 07:01:27.799721 (+ 15us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:26.829047 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:26.829063 (+ 16us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:26.829064 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:01:26.829064 (+ 0us) tablet.cc:660] Decoding operations
0808 07:01:26.829078 (+ 14us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:26.829083 (+ 5us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:26.829084 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:26.829094 (+ 10us) tablet.cc:699] Row locks acquired
0808 07:01:26.829095 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:01:27.799445 (+970350us) op_driver.cc:444] REPLICATION: finished
0808 07:01:27.799598 (+ 153us) write_op.cc:301] APPLY: starting
0808 07:01:27.799602 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:27.799607 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:27.799608 (+ 1us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:27.799623 (+ 15us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:27.799628 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:27.799632 (+ 4us) write_op.cc:312] APPLY: finished
0808 07:01:27.799653 (+ 21us) log.cc:842] Serialized 22 byte log entry
0808 07:01:27.799660 (+ 7us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:27.799663 (+ 3us) write_op.cc:454] Released schema lock
0808 07:01:27.799664 (+ 1us) write_op.cc:341] FINISH: Updating metrics
0808 07:01:27.799683 (+ 19us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:27.799683 (+ 0us) write_op.cc:454] Released schema lock
Related trace 'op':
0808 07:01:27.799408 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:27.799422 (+ 14us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:27.799423 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:01:27.799423 (+ 0us) tablet.cc:660] Decoding operations
0808 07:01:27.799436 (+ 13us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:27.799454 (+ 18us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:27.799455 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:27.799465 (+ 10us) tablet.cc:699] Row locks acquired
0808 07:01:27.799465 (+ 0us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":58,"threads_started":1,"wal-append.queue_time_us":85,"child_traces":[["op",{"apply.queue_time_us":138,"apply.run_cpu_time_us":100,"apply.run_wall_time_us":100,"num_ops":1,"partition_lock_wait_us":2,"prepare.queue_time_us":42,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":67,"replication_time_us":1067640,"thread_start_us":65,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":16,"prepare.queue_time_us":162,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":76,"thread_start_us":75,"threads_started":1}]]}
I20260808 07:01:28.083925 18488 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:28.084066 18488 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:28.084347 18488 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:28.084570 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:28.084544 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:28.084652 18503 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=3, attempt_no=0}) took 1395 ms. Trace:
I20260808 07:01:28.084812 18041 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:28.084757 18503 rpcz_store.cc:276] 0808 07:01:26.689071 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:26.689135 (+ 64us) service_pool.cc:227] Handling call
0808 07:01:28.084642 (+1395507us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:26.716872 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:26.716894 (+ 22us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:26.716895 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:01:26.716896 (+ 1us) tablet.cc:660] Decoding operations
0808 07:01:26.716915 (+ 19us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:26.716918 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:26.716919 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:26.716930 (+ 11us) tablet.cc:699] Row locks acquired
0808 07:01:26.716930 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:01:26.716935 (+ 5us) write_op.cc:270] Start()
0808 07:01:27.666084 (+949149us) write_op.cc:276] Timestamp: P: 1786172486716908 usec, L: 0
0808 07:01:27.666173 (+ 89us) op_driver.cc:331] REPLICATION: starting
0808 07:01:27.666240 (+ 67us) log.cc:842] Serialized 354 byte log entry
0808 07:01:28.084494 (+418254us) op_driver.cc:444] REPLICATION: finished
0808 07:01:28.084526 (+ 32us) write_op.cc:301] APPLY: starting
0808 07:01:28.084529 (+ 3us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:28.084535 (+ 6us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:28.084536 (+ 1us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:28.084549 (+ 13us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:28.084555 (+ 6us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:28.084556 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:01:28.084582 (+ 26us) log.cc:842] Serialized 22 byte log entry
0808 07:01:28.084617 (+ 35us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:28.084620 (+ 3us) write_op.cc:454] Released schema lock
0808 07:01:28.084620 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":21,"num_ops":1,"prepare.queue_time_us":27578,"prepare.run_cpu_time_us":345,"prepare.run_wall_time_us":949398,"replication_time_us":418314}]]}
I20260808 07:01:28.084947 18488 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:28.482153 18488 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:28.482259 18488 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:28.482466 18488 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:28.482652 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:28.482704 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:28.483202 18042 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:28.483356 18488 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 07:01:28.483357 18477 scanner-internal.cc:458] Time spent opening tablet: real 0.722s user 0.000s sys 0.000s
W20260808 07:01:28.483394 18478 scanner-internal.cc:458] Time spent opening tablet: real 0.722s user 0.000s sys 0.000s
I20260808 07:01:28.780206 18487 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:28.780350 18487 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:28.780532 18487 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 910bd273924d4cb0adc0f7b65e454f19 (127.15.250.129:45605), 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273)
I20260808 07:01:28.780745 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:28.780745 18112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "910bd273924d4cb0adc0f7b65e454f19" is_pre_election: true
I20260808 07:01:28.781132 18322 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 221e532665ab4c7f9585be8ed849b6a9; no voters: 03f850c83dff43fdade5214b4dbee7a1, 910bd273924d4cb0adc0f7b65e454f19
I20260808 07:01:29.375736 18510 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:29.375876 18510 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:29.376127 18510 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:29.376286 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:29.376361 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:29.376710 18041 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:29.376843 18513 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=5, attempt_no=0}) took 1291 ms. Trace:
I20260808 07:01:29.376959 18510 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:29.376920 18513 rpcz_store.cc:276] 0808 07:01:28.085456 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:28.085483 (+ 27us) service_pool.cc:227] Handling call
0808 07:01:29.376833 (+1291350us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:28.331562 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:28.331578 (+ 16us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:28.331578 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:01:28.331579 (+ 1us) tablet.cc:660] Decoding operations
0808 07:01:28.331593 (+ 14us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:28.331595 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:28.331596 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:28.331608 (+ 12us) tablet.cc:699] Row locks acquired
0808 07:01:28.331609 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:01:28.331614 (+ 5us) write_op.cc:270] Start()
0808 07:01:29.039708 (+708094us) write_op.cc:276] Timestamp: P: 1786172488331567 usec, L: 0
0808 07:01:29.039709 (+ 1us) op_driver.cc:331] REPLICATION: starting
0808 07:01:29.039819 (+ 110us) log.cc:842] Serialized 354 byte log entry
0808 07:01:29.376371 (+336552us) op_driver.cc:444] REPLICATION: finished
0808 07:01:29.376470 (+ 99us) write_op.cc:301] APPLY: starting
0808 07:01:29.376473 (+ 3us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:29.376497 (+ 24us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:29.376500 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:29.376513 (+ 13us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:29.376519 (+ 6us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:29.376520 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:01:29.376539 (+ 19us) log.cc:842] Serialized 22 byte log entry
0808 07:01:29.376655 (+ 116us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:29.376658 (+ 3us) write_op.cc:454] Released schema lock
0808 07:01:29.376659 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":82,"num_ops":1,"prepare.queue_time_us":245938,"prepare.run_cpu_time_us":238,"prepare.run_wall_time_us":708303,"raft.queue_time_us":32,"raft.run_cpu_time_us":71,"raft.run_wall_time_us":71,"replication_time_us":336656,"thread_start_us":46,"threads_started":1}]]}
W20260808 07:01:29.377084 18478 scanner-internal.cc:458] Time spent opening tablet: real 0.743s user 0.000s sys 0.000s
W20260808 07:01:29.377334 18477 scanner-internal.cc:458] Time spent opening tablet: real 0.744s user 0.000s sys 0.000s
I20260808 07:01:29.572674 18510 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:29.572795 18510 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:29.573045 18510 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:29.573290 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:29.573290 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:29.573436 18513 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=6, attempt_no=0}) took 1089 ms. Trace:
I20260808 07:01:29.573524 18513 rpcz_store.cc:276] 0808 07:01:28.483462 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:28.483510 (+ 48us) service_pool.cc:227] Handling call
0808 07:01:29.573426 (+1089916us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:29.039876 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:29.039892 (+ 16us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:29.039892 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:01:29.039893 (+ 1us) tablet.cc:660] Decoding operations
0808 07:01:29.039907 (+ 14us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:29.039909 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:29.039910 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:29.039920 (+ 10us) tablet.cc:699] Row locks acquired
0808 07:01:29.039921 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:01:29.039926 (+ 5us) write_op.cc:270] Start()
0808 07:01:29.425024 (+385098us) write_op.cc:276] Timestamp: P: 1786172489039860 usec, L: 0
0808 07:01:29.425025 (+ 1us) op_driver.cc:331] REPLICATION: starting
0808 07:01:29.425075 (+ 50us) log.cc:842] Serialized 354 byte log entry
0808 07:01:29.573281 (+148206us) op_driver.cc:444] REPLICATION: finished
0808 07:01:29.573314 (+ 33us) write_op.cc:301] APPLY: starting
0808 07:01:29.573318 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:29.573323 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:29.573325 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:29.573338 (+ 13us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:29.573344 (+ 6us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:29.573347 (+ 3us) write_op.cc:312] APPLY: finished
0808 07:01:29.573369 (+ 22us) log.cc:842] Serialized 22 byte log entry
0808 07:01:29.573403 (+ 34us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:29.573406 (+ 3us) write_op.cc:454] Released schema lock
0808 07:01:29.573406 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":21,"num_ops":1,"prepare.queue_time_us":556256,"prepare.run_cpu_time_us":179,"prepare.run_wall_time_us":385247,"raft.queue_time_us":30,"raft.run_cpu_time_us":52,"raft.run_wall_time_us":52,"replication_time_us":148252}]]}
I20260808 07:01:29.573921 18042 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:29.574019 18510 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:29.720651 18487 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:29.798988 18510 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:29.799110 18510 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:29.799350 18510 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:29.799557 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:29.799615 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:29.799911 18042 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:29.800128 18510 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:30.345948 18487 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:30.346051 18487 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:30.346493 18487 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 910bd273924d4cb0adc0f7b65e454f19 (127.15.250.129:45605), 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273)
I20260808 07:01:30.346890 18112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "910bd273924d4cb0adc0f7b65e454f19" is_pre_election: true
I20260808 07:01:30.346899 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:30.347141 18322 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 221e532665ab4c7f9585be8ed849b6a9; no voters: 03f850c83dff43fdade5214b4dbee7a1, 910bd273924d4cb0adc0f7b65e454f19
I20260808 07:01:30.524487 18487 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:30.902549 18521 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:30.902678 18521 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:30.902943 18521 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:30.903083 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:30.903119 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:30.903452 18524 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=8, attempt_no=0}) took 1329 ms. Trace:
I20260808 07:01:30.903513 18524 rpcz_store.cc:276] 0808 07:01:29.574228 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:29.574317 (+ 89us) service_pool.cc:227] Handling call
0808 07:01:30.903443 (+1329126us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:29.743446 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:29.743462 (+ 16us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:29.743463 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:01:29.743463 (+ 0us) tablet.cc:660] Decoding operations
0808 07:01:29.743477 (+ 14us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:29.743480 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:29.743482 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:29.743492 (+ 10us) tablet.cc:699] Row locks acquired
0808 07:01:29.743493 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:01:29.743498 (+ 5us) write_op.cc:270] Start()
0808 07:01:30.719599 (+976101us) write_op.cc:276] Timestamp: P: 1786172489743473 usec, L: 0
0808 07:01:30.719601 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:01:30.719705 (+ 104us) log.cc:842] Serialized 354 byte log entry
0808 07:01:30.903085 (+183380us) op_driver.cc:444] REPLICATION: finished
0808 07:01:30.903265 (+ 180us) write_op.cc:301] APPLY: starting
0808 07:01:30.903269 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:30.903274 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:30.903277 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:30.903291 (+ 14us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:30.903296 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:30.903297 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:01:30.903318 (+ 21us) log.cc:842] Serialized 22 byte log entry
0808 07:01:30.903419 (+ 101us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:30.903422 (+ 3us) write_op.cc:454] Released schema lock
0808 07:01:30.903423 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":165,"num_ops":1,"prepare.queue_time_us":169015,"prepare.run_cpu_time_us":255,"prepare.run_wall_time_us":976327,"raft.queue_time_us":83,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":98,"replication_time_us":183480,"spinlock_wait_cycles":384,"thread_start_us":70,"threads_started":1}]]}
I20260808 07:01:30.903666 18042 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:30.903779 18521 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 07:01:30.903795 18477 scanner-internal.cc:458] Time spent opening tablet: real 0.954s user 0.000s sys 0.000s
W20260808 07:01:30.903910 18478 scanner-internal.cc:458] Time spent opening tablet: real 0.954s user 0.000s sys 0.000s
I20260808 07:01:31.396610 18521 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:31.396739 18521 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:31.396965 18521 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:31.397198 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:31.397280 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:31.397367 18524 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=9, attempt_no=0}) took 1597 ms. Trace:
I20260808 07:01:31.397414 18524 rpcz_store.cc:276] 0808 07:01:29.800154 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:29.800179 (+ 25us) service_pool.cc:227] Handling call
0808 07:01:31.397362 (+1597183us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:30.719782 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:30.719798 (+ 16us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:30.719798 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:01:30.719799 (+ 1us) tablet.cc:660] Decoding operations
0808 07:01:30.719813 (+ 14us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:30.719815 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:30.719816 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:30.719828 (+ 12us) tablet.cc:699] Row locks acquired
0808 07:01:30.719828 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:01:30.719833 (+ 5us) write_op.cc:270] Start()
0808 07:01:30.993929 (+274096us) write_op.cc:276] Timestamp: P: 1786172490719823 usec, L: 0
0808 07:01:30.993930 (+ 1us) op_driver.cc:331] REPLICATION: starting
0808 07:01:30.993999 (+ 69us) log.cc:842] Serialized 354 byte log entry
0808 07:01:31.397225 (+403226us) op_driver.cc:444] REPLICATION: finished
0808 07:01:31.397263 (+ 38us) write_op.cc:301] APPLY: starting
0808 07:01:31.397268 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:31.397273 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:31.397275 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:31.397288 (+ 13us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:31.397294 (+ 6us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:31.397294 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:01:31.397318 (+ 24us) log.cc:842] Serialized 22 byte log entry
0808 07:01:31.397339 (+ 21us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:31.397342 (+ 3us) write_op.cc:454] Released schema lock
0808 07:01:31.397343 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":23,"num_ops":1,"prepare.queue_time_us":919450,"prepare.run_cpu_time_us":173,"prepare.run_wall_time_us":274241,"replication_time_us":403290}]]}
I20260808 07:01:31.397946 18042 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:31.667484 18487 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:31.667598 18487 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:31.667826 18487 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 910bd273924d4cb0adc0f7b65e454f19 (127.15.250.129:45605), 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273)
I20260808 07:01:31.668022 18111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "910bd273924d4cb0adc0f7b65e454f19" is_pre_election: true
I20260808 07:01:31.668041 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:31.668390 18322 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 221e532665ab4c7f9585be8ed849b6a9; no voters: 03f850c83dff43fdade5214b4dbee7a1, 910bd273924d4cb0adc0f7b65e454f19
I20260808 07:01:31.843441 18521 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 07:01:31.844700 18478 scanner-internal.cc:458] Time spent opening tablet: real 0.791s user 0.000s sys 0.000s
W20260808 07:01:31.844723 18477 scanner-internal.cc:458] Time spent opening tablet: real 0.791s user 0.000s sys 0.000s
I20260808 07:01:32.036845 18521 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:32.036962 18521 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:32.037175 18521 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:32.037436 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:32.037446 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:32.037845 18042 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:32.038112 18521 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:33.011999 18529 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:33.012126 18529 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:33.012506 18529 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:33.012610 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:33.012596 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:33.012863 18041 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:33.012980 18529 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:33.013144 18532 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=12, attempt_no=0}) took 1168 ms. Trace:
I20260808 07:01:33.013202 18532 rpcz_store.cc:276] 0808 07:01:31.844837 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:31.844891 (+ 54us) service_pool.cc:227] Handling call
0808 07:01:33.013134 (+1168243us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:31.845037 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:31.845048 (+ 11us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:31.845049 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:01:31.845049 (+ 0us) tablet.cc:660] Decoding operations
0808 07:01:31.845058 (+ 9us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:31.845059 (+ 1us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:31.845060 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:31.845067 (+ 7us) tablet.cc:699] Row locks acquired
0808 07:01:31.845067 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:01:31.845071 (+ 4us) write_op.cc:270] Start()
0808 07:01:32.838170 (+993099us) write_op.cc:276] Timestamp: P: 1786172491845041 usec, L: 0
0808 07:01:32.838171 (+ 1us) op_driver.cc:331] REPLICATION: starting
0808 07:01:32.838279 (+ 108us) log.cc:842] Serialized 354 byte log entry
0808 07:01:33.012875 (+174596us) op_driver.cc:444] REPLICATION: finished
0808 07:01:33.013009 (+ 134us) write_op.cc:301] APPLY: starting
0808 07:01:33.013013 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:33.013020 (+ 7us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:33.013022 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:33.013035 (+ 13us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:33.013048 (+ 13us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:33.013049 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:01:33.013069 (+ 20us) log.cc:842] Serialized 22 byte log entry
0808 07:01:33.013109 (+ 40us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:33.013112 (+ 3us) write_op.cc:454] Released schema lock
0808 07:01:33.013112 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":119,"num_ops":1,"prepare.queue_time_us":37,"prepare.run_cpu_time_us":222,"prepare.run_wall_time_us":993292,"replication_time_us":174699,"thread_start_us":71,"threads_started":1}]]}
I20260808 07:01:33.013284 18208 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60832 (request call id 28) took 1017 ms. Trace:
I20260808 07:01:33.013329 18207 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60832 (request call id 27) took 1018 ms. Trace:
I20260808 07:01:33.013365 18207 rpcz_store.cc:276] 0808 07:01:31.995316 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:31.995378 (+ 62us) service_pool.cc:227] Handling call
0808 07:01:31.995448 (+ 70us) tablet_service.cc:2780] Created scanner 3b4d5a673fb24fe7ba226a4b9228c138 for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is f452ff33918e478182e76d5f113541c3
0808 07:01:31.995491 (+ 43us) tablet_service.cc:2920] Creating iterator
0808 07:01:31.995501 (+ 10us) tablet_service.cc:3297] Waiting safe time to advance
0808 07:01:32.838244 (+842743us) tablet_service.cc:3304] Waiting for operations to commit
0808 07:01:33.013194 (+174950us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1017690 microseconds
0808 07:01:33.013209 (+ 15us) tablet_service.cc:2945] Iterator created
0808 07:01:33.013258 (+ 49us) tablet_service.cc:2967] Iterator init: OK
0808 07:01:33.013261 (+ 3us) tablet_service.cc:3010] has_more: true
0808 07:01:33.013266 (+ 5us) tablet_service.cc:3027] Continuing scan request
0808 07:01:33.013278 (+ 12us) tablet_service.cc:3091] Found scanner 3b4d5a673fb24fe7ba226a4b9228c138 for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is f452ff33918e478182e76d5f113541c3
0808 07:01:33.013327 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":22016}
I20260808 07:01:33.013360 18208 rpcz_store.cc:276] 0808 07:01:31.995344 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:31.995392 (+ 48us) service_pool.cc:227] Handling call
0808 07:01:31.995445 (+ 53us) tablet_service.cc:2780] Created scanner 0458b1ef3982410dab272a3957ca2123 for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is 25081a96cb074063bc66da647a3d0072
0808 07:01:31.995489 (+ 44us) tablet_service.cc:2920] Creating iterator
0808 07:01:31.995501 (+ 12us) tablet_service.cc:3297] Waiting safe time to advance
0808 07:01:32.838228 (+842727us) tablet_service.cc:3304] Waiting for operations to commit
0808 07:01:33.013120 (+174892us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1017616 microseconds
0808 07:01:33.013140 (+ 20us) tablet_service.cc:2945] Iterator created
0808 07:01:33.013188 (+ 48us) tablet_service.cc:2967] Iterator init: OK
0808 07:01:33.013192 (+ 4us) tablet_service.cc:3010] has_more: true
0808 07:01:33.013198 (+ 6us) tablet_service.cc:3027] Continuing scan request
0808 07:01:33.013214 (+ 16us) tablet_service.cc:3091] Found scanner 0458b1ef3982410dab272a3957ca2123 for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is 25081a96cb074063bc66da647a3d0072
0808 07:01:33.013278 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":8960}
W20260808 07:01:33.013844 18478 scanner-internal.cc:458] Time spent opening tablet: real 1.019s user 0.000s sys 0.000s
W20260808 07:01:33.013854 18477 scanner-internal.cc:458] Time spent opening tablet: real 1.019s user 0.000s sys 0.000s
I20260808 07:01:33.452596 18529 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:33.452755 18529 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:33.452991 18529 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:33.453135 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:33.453234 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:33.453294 18532 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=13, attempt_no=0}) took 1415 ms. Trace:
I20260808 07:01:33.453383 18532 rpcz_store.cc:276] 0808 07:01:32.038059 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:32.038084 (+ 25us) service_pool.cc:227] Handling call
0808 07:01:33.453289 (+1415205us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:32.838335 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:32.838348 (+ 13us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:32.838349 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:01:32.838349 (+ 0us) tablet.cc:660] Decoding operations
0808 07:01:32.838360 (+ 11us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:32.838362 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:32.838363 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:32.838372 (+ 9us) tablet.cc:699] Row locks acquired
0808 07:01:32.838373 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:01:32.838377 (+ 4us) write_op.cc:270] Start()
0808 07:01:33.003498 (+165121us) write_op.cc:276] Timestamp: P: 1786172492838352 usec, L: 0
0808 07:01:33.003501 (+ 3us) op_driver.cc:331] REPLICATION: starting
0808 07:01:33.003570 (+ 69us) log.cc:842] Serialized 354 byte log entry
0808 07:01:33.453142 (+449572us) op_driver.cc:444] REPLICATION: finished
0808 07:01:33.453186 (+ 44us) write_op.cc:301] APPLY: starting
0808 07:01:33.453190 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:33.453197 (+ 7us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:33.453199 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:33.453211 (+ 12us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:33.453217 (+ 6us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:33.453218 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:01:33.453237 (+ 19us) log.cc:842] Serialized 22 byte log entry
0808 07:01:33.453255 (+ 18us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:33.453258 (+ 3us) write_op.cc:454] Released schema lock
0808 07:01:33.453258 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":32,"num_ops":1,"prepare.queue_time_us":800100,"prepare.run_cpu_time_us":175,"prepare.run_wall_time_us":165259,"replication_time_us":449638}]]}
I20260808 07:01:33.453860 18042 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:33.453962 18529 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:33.546511 18487 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:33.546610 18388 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.15.250.130:38799 (request call id 530) took 1878 ms. Trace:
I20260808 07:01:33.546676 18388 rpcz_store.cc:276] 0808 07:01:31.668349 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:31.668419 (+ 70us) service_pool.cc:227] Handling call
0808 07:01:31.668462 (+ 43us) raft_consensus.cc:1457] Updating replica for 3 ops
0808 07:01:31.668477 (+ 15us) raft_consensus.cc:1520] Early marking committed up to index 10
0808 07:01:31.668574 (+ 97us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0808 07:01:31.668618 (+ 44us) write_op.cc:270] Start()
0808 07:01:32.135726 (+467108us) write_op.cc:276] Timestamp: P: 1786172490719823 usec, L: 0
0808 07:01:32.135818 (+ 92us) write_op.cc:270] Start()
0808 07:01:32.606913 (+471095us) write_op.cc:276] Timestamp: P: 1786172490994052 usec, L: 0
0808 07:01:32.607005 (+ 92us) write_op.cc:270] Start()
0808 07:01:33.546113 (+939108us) write_op.cc:276] Timestamp: P: 1786172491398131 usec, L: 0
0808 07:01:33.546340 (+ 227us) log.cc:842] Serialized 1062 byte log entry
0808 07:01:33.546402 (+ 62us) raft_consensus.cc:1631] Marking committed up to 11
0808 07:01:33.546452 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 07:01:33.546467 (+ 15us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 07:01:33.546586 (+ 119us) raft_consensus.cc:1671] finished
0808 07:01:33.546587 (+ 1us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 07:01:33.546600 (+ 13us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:32.135819 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:32.135835 (+ 16us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:32.135835 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:01:32.135835 (+ 0us) tablet.cc:660] Decoding operations
0808 07:01:32.135851 (+ 16us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:32.135855 (+ 4us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:32.135856 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:32.135872 (+ 16us) tablet.cc:699] Row locks acquired
0808 07:01:32.135873 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:01:33.546409 (+1410536us) op_driver.cc:444] REPLICATION: finished
0808 07:01:33.546511 (+ 102us) write_op.cc:301] APPLY: starting
0808 07:01:33.546515 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:33.546519 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:33.546520 (+ 1us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:33.546532 (+ 12us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:33.546537 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:33.546539 (+ 2us) write_op.cc:312] APPLY: finished
0808 07:01:33.546558 (+ 19us) log.cc:842] Serialized 22 byte log entry
0808 07:01:33.546565 (+ 7us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:33.546567 (+ 2us) write_op.cc:454] Released schema lock
0808 07:01:33.546568 (+ 1us) write_op.cc:341] FINISH: Updating metrics
0808 07:01:33.546584 (+ 16us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:33.546584 (+ 0us) write_op.cc:454] Released schema lock
Related trace 'op':
0808 07:01:32.607005 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:32.607020 (+ 15us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:32.607021 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:01:32.607021 (+ 0us) tablet.cc:660] Decoding operations
0808 07:01:32.607035 (+ 14us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:32.607040 (+ 5us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:32.607041 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:32.607057 (+ 16us) tablet.cc:699] Row locks acquired
0808 07:01:32.607057 (+ 0us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0808 07:01:33.546352 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:33.546370 (+ 18us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:33.546371 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:01:33.546371 (+ 0us) tablet.cc:660] Decoding operations
0808 07:01:33.546392 (+ 21us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:33.546395 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:33.546396 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:33.546405 (+ 9us) tablet.cc:699] Row locks acquired
0808 07:01:33.546406 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":52,"threads_started":1,"wal-append.queue_time_us":77,"child_traces":[["op",{"apply.queue_time_us":92,"apply.run_cpu_time_us":85,"apply.run_wall_time_us":85,"num_ops":1,"partition_lock_wait_us":2,"prepare.queue_time_us":43,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":74,"replication_time_us":1877787,"thread_start_us":36,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":1,"prepare.queue_time_us":43,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":71}],["op",{"num_ops":1,"partition_lock_wait_us":1,"prepare.queue_time_us":184,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":69,"thread_start_us":111,"threads_started":1}]]}
I20260808 07:01:34.311232 18539 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:34.311359 18539 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:34.311651 18539 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:34.311887 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:34.311913 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:34.312204 18208 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60832 (request call id 31) took 1147 ms. Trace:
I20260808 07:01:34.312213 18207 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60832 (request call id 32) took 1147 ms. Trace:
I20260808 07:01:34.312336 18207 rpcz_store.cc:276] 0808 07:01:33.164508 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:33.164577 (+ 69us) service_pool.cc:227] Handling call
0808 07:01:33.164664 (+ 87us) tablet_service.cc:2780] Created scanner 170ba1fe9cea454fb8a5413a49107d11 for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is 3108833339c44d058b30766dc5c62a11
0808 07:01:33.164700 (+ 36us) tablet_service.cc:2920] Creating iterator
0808 07:01:33.164710 (+ 10us) tablet_service.cc:3297] Waiting safe time to advance
0808 07:01:33.986499 (+821789us) tablet_service.cc:3304] Waiting for operations to commit
0808 07:01:34.312082 (+325583us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1147368 microseconds
0808 07:01:34.312097 (+ 15us) tablet_service.cc:2945] Iterator created
0808 07:01:34.312146 (+ 49us) tablet_service.cc:2967] Iterator init: OK
0808 07:01:34.312148 (+ 2us) tablet_service.cc:3010] has_more: true
0808 07:01:34.312153 (+ 5us) tablet_service.cc:3027] Continuing scan request
0808 07:01:34.312166 (+ 13us) tablet_service.cc:3091] Found scanner 170ba1fe9cea454fb8a5413a49107d11 for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is 3108833339c44d058b30766dc5c62a11
0808 07:01:34.312208 (+ 42us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":33280}
I20260808 07:01:34.312291 18208 rpcz_store.cc:276] 0808 07:01:33.164464 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:33.164577 (+ 113us) service_pool.cc:227] Handling call
0808 07:01:33.164642 (+ 65us) tablet_service.cc:2780] Created scanner a6eee41fae674c8bb598f73b037015bd for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is 1d7b704304744535974a7f49ea10bd3b
0808 07:01:33.164699 (+ 57us) tablet_service.cc:2920] Creating iterator
0808 07:01:33.164710 (+ 11us) tablet_service.cc:3297] Waiting safe time to advance
0808 07:01:33.986502 (+821792us) tablet_service.cc:3304] Waiting for operations to commit
0808 07:01:34.312025 (+325523us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1147311 microseconds
0808 07:01:34.312043 (+ 18us) tablet_service.cc:2945] Iterator created
0808 07:01:34.312095 (+ 52us) tablet_service.cc:2967] Iterator init: OK
0808 07:01:34.312098 (+ 3us) tablet_service.cc:3010] has_more: true
0808 07:01:34.312104 (+ 6us) tablet_service.cc:3027] Continuing scan request
0808 07:01:34.312126 (+ 22us) tablet_service.cc:3091] Found scanner a6eee41fae674c8bb598f73b037015bd for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is 1d7b704304744535974a7f49ea10bd3b
0808 07:01:34.312199 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":3200}
I20260808 07:01:34.312456 18041 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:34.312601 18543 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=14, attempt_no=0}) took 1298 ms. Trace:
W20260808 07:01:34.312690 18478 scanner-internal.cc:458] Time spent opening tablet: real 1.149s user 0.000s sys 0.000s
I20260808 07:01:34.312644 18543 rpcz_store.cc:276] 0808 07:01:33.014139 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:33.014158 (+ 19us) service_pool.cc:227] Handling call
0808 07:01:34.312596 (+1298438us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:33.014293 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:33.014302 (+ 9us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:33.014303 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:01:33.014303 (+ 0us) tablet.cc:660] Decoding operations
0808 07:01:33.014311 (+ 8us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:33.014313 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:33.014313 (+ 0us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:33.014322 (+ 9us) tablet.cc:699] Row locks acquired
0808 07:01:33.014322 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:01:33.014327 (+ 5us) write_op.cc:270] Start()
0808 07:01:33.986420 (+972093us) write_op.cc:276] Timestamp: P: 1786172493014288 usec, L: 0
0808 07:01:33.986422 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:01:33.986499 (+ 77us) log.cc:842] Serialized 354 byte log entry
0808 07:01:34.311775 (+325276us) op_driver.cc:444] REPLICATION: finished
0808 07:01:34.311912 (+ 137us) write_op.cc:301] APPLY: starting
0808 07:01:34.311916 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:34.311921 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:34.311924 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:34.311952 (+ 28us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:34.311957 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:34.311958 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:01:34.311975 (+ 17us) log.cc:842] Serialized 22 byte log entry
0808 07:01:34.312495 (+ 520us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:34.312497 (+ 2us) write_op.cc:454] Released schema lock
0808 07:01:34.312498 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":124,"num_ops":1,"prepare.queue_time_us":36,"prepare.run_cpu_time_us":200,"prepare.run_wall_time_us":972267,"raft.queue_time_us":18,"raft.run_cpu_time_us":59,"raft.run_wall_time_us":54,"replication_time_us":325348,"thread_start_us":54,"threads_started":1}]]}
W20260808 07:01:34.312690 18477 scanner-internal.cc:458] Time spent opening tablet: real 1.149s user 0.000s sys 0.000s
I20260808 07:01:34.558104 18487 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:34.558202 18487 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:34.558239 18388 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.15.250.130:38799 (request call id 531) took 1011 ms. Trace:
I20260808 07:01:34.558390 18487 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 910bd273924d4cb0adc0f7b65e454f19 (127.15.250.129:45605), 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273)
I20260808 07:01:34.558307 18388 rpcz_store.cc:276] 0808 07:01:33.547177 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:33.547197 (+ 20us) service_pool.cc:227] Handling call
0808 07:01:33.547232 (+ 35us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 07:01:33.547248 (+ 16us) raft_consensus.cc:1520] Early marking committed up to index 13
0808 07:01:33.547350 (+ 102us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 07:01:33.547396 (+ 46us) write_op.cc:270] Start()
0808 07:01:34.220491 (+673095us) write_op.cc:276] Timestamp: P: 1786172491845041 usec, L: 0
0808 07:01:34.220647 (+ 156us) write_op.cc:270] Start()
0808 07:01:34.557762 (+337115us) write_op.cc:276] Timestamp: P: 1786172492838352 usec, L: 0
0808 07:01:34.557864 (+ 102us) log.cc:842] Serialized 708 byte log entry
0808 07:01:34.557954 (+ 90us) raft_consensus.cc:1631] Marking committed up to 15
0808 07:01:34.558053 (+ 99us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 07:01:34.558067 (+ 14us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 07:01:34.558214 (+ 147us) raft_consensus.cc:1671] finished
0808 07:01:34.558214 (+ 0us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 07:01:34.558228 (+ 14us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:34.220682 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:34.220697 (+ 15us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:34.220697 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:01:34.220698 (+ 1us) tablet.cc:660] Decoding operations
0808 07:01:34.220710 (+ 12us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:34.220715 (+ 5us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:34.220716 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:34.220725 (+ 9us) tablet.cc:699] Row locks acquired
0808 07:01:34.220726 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:01:34.557960 (+337234us) op_driver.cc:444] REPLICATION: finished
0808 07:01:34.558047 (+ 87us) write_op.cc:301] APPLY: starting
0808 07:01:34.558051 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:34.558056 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:34.558057 (+ 1us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:34.558069 (+ 12us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:34.558074 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:34.558077 (+ 3us) write_op.cc:312] APPLY: finished
0808 07:01:34.558093 (+ 16us) log.cc:842] Serialized 22 byte log entry
0808 07:01:34.558099 (+ 6us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:34.558101 (+ 2us) write_op.cc:454] Released schema lock
0808 07:01:34.558102 (+ 1us) write_op.cc:341] FINISH: Updating metrics
0808 07:01:34.558115 (+ 13us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:34.558116 (+ 1us) write_op.cc:454] Released schema lock
Related trace 'op':
0808 07:01:34.557855 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:34.557870 (+ 15us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:34.557871 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:01:34.557871 (+ 0us) tablet.cc:660] Decoding operations
0808 07:01:34.557887 (+ 16us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:34.557890 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:34.557891 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:34.557901 (+ 10us) tablet.cc:699] Row locks acquired
0808 07:01:34.557902 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:01:34.558018 (+ 116us) op_driver.cc:444] REPLICATION: finished
0808 07:01:34.558094 (+ 76us) write_op.cc:301] APPLY: starting
0808 07:01:34.558094 (+ 0us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:34.558096 (+ 2us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:34.558097 (+ 1us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:34.558100 (+ 3us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:34.558101 (+ 1us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:34.558102 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:01:34.558107 (+ 5us) log.cc:842] Serialized 22 byte log entry
0808 07:01:34.558109 (+ 2us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:34.558111 (+ 2us) write_op.cc:454] Released schema lock
0808 07:01:34.558112 (+ 1us) write_op.cc:341] FINISH: Updating metrics
0808 07:01:34.558116 (+ 4us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:34.558116 (+ 0us) write_op.cc:454] Released schema lock
Metrics: {"thread_start_us":80,"threads_started":1,"wal-append.queue_time_us":135,"child_traces":[["op",{"apply.queue_time_us":77,"apply.run_cpu_time_us":78,"apply.run_wall_time_us":79,"mutex_wait_us":3,"num_ops":1,"partition_lock_wait_us":2,"prepare.queue_time_us":140,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":86,"replication_time_us":1010561,"thread_start_us":137,"threads_started":2}],["op",{"apply.queue_time_us":71,"apply.run_cpu_time_us":30,"apply.run_wall_time_us":29,"num_ops":1,"partition_lock_wait_us":1,"prepare.queue_time_us":36,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":76,"replication_time_us":337369,"thread_start_us":31,"threads_started":1}]]}
I20260808 07:01:34.558640 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:34.558822 18111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "910bd273924d4cb0adc0f7b65e454f19" is_pre_election: true
I20260808 07:01:34.559267 18322 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 221e532665ab4c7f9585be8ed849b6a9; no voters: 03f850c83dff43fdade5214b4dbee7a1, 910bd273924d4cb0adc0f7b65e454f19
I20260808 07:01:34.696421 18539 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:34.697085 18543 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=15, attempt_no=0}) took 1242 ms. Trace:
I20260808 07:01:34.697170 18543 rpcz_store.cc:276] 0808 07:01:33.454108 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:33.454131 (+ 23us) service_pool.cc:227] Handling call
0808 07:01:34.697070 (+1242939us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:33.986568 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:33.986586 (+ 18us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:33.986586 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:01:33.986587 (+ 1us) tablet.cc:660] Decoding operations
0808 07:01:33.986599 (+ 12us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:33.986603 (+ 4us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:33.986604 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:33.986614 (+ 10us) tablet.cc:699] Row locks acquired
0808 07:01:33.986615 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:01:33.986621 (+ 6us) write_op.cc:270] Start()
0808 07:01:34.122746 (+136125us) write_op.cc:276] Timestamp: P: 1786172493986574 usec, L: 0
0808 07:01:34.122748 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:01:34.122814 (+ 66us) log.cc:842] Serialized 354 byte log entry
0808 07:01:34.696891 (+574077us) op_driver.cc:444] REPLICATION: finished
0808 07:01:34.696944 (+ 53us) write_op.cc:301] APPLY: starting
0808 07:01:34.696948 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:34.696959 (+ 11us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:34.696962 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:34.696991 (+ 29us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:34.696996 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:34.696997 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:01:34.697020 (+ 23us) log.cc:842] Serialized 22 byte log entry
0808 07:01:34.697043 (+ 23us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:34.697046 (+ 3us) write_op.cc:454] Released schema lock
0808 07:01:34.697047 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":39,"num_ops":1,"prepare.queue_time_us":532261,"prepare.run_cpu_time_us":229,"prepare.run_wall_time_us":136305,"replication_time_us":574138}]]}
I20260808 07:01:34.761564 18539 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:34.761682 18539 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:34.761921 18539 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:34.762148 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:34.762313 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:34.762771 18041 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:34.762894 18539 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:35.803112 18539 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:35.803228 18539 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:35.803478 18539 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:35.803653 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:35.803674 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:35.803977 18041 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:35.804138 18539 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:35.804584 18554 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=17, attempt_no=0}) took 1106 ms. Trace:
I20260808 07:01:35.804661 18554 rpcz_store.cc:276] 0808 07:01:34.697829 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:34.697870 (+ 41us) service_pool.cc:227] Handling call
0808 07:01:35.804571 (+1106701us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:34.698014 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:34.698024 (+ 10us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:34.698024 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:01:34.698025 (+ 1us) tablet.cc:660] Decoding operations
0808 07:01:34.698033 (+ 8us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:34.698034 (+ 1us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:34.698035 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:34.698042 (+ 7us) tablet.cc:699] Row locks acquired
0808 07:01:34.698042 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:01:34.698049 (+ 7us) write_op.cc:270] Start()
0808 07:01:35.129153 (+431104us) write_op.cc:276] Timestamp: P: 1786172494697994 usec, L: 0
0808 07:01:35.129155 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:01:35.129233 (+ 78us) log.cc:842] Serialized 354 byte log entry
0808 07:01:35.804245 (+675012us) op_driver.cc:444] REPLICATION: finished
0808 07:01:35.804393 (+ 148us) write_op.cc:301] APPLY: starting
0808 07:01:35.804399 (+ 6us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:35.804406 (+ 7us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:35.804411 (+ 5us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:35.804428 (+ 17us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:35.804436 (+ 8us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:35.804437 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:01:35.804470 (+ 33us) log.cc:842] Serialized 22 byte log entry
0808 07:01:35.804540 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:35.804543 (+ 3us) write_op.cc:454] Released schema lock
0808 07:01:35.804543 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":125,"num_ops":1,"prepare.queue_time_us":31,"prepare.run_cpu_time_us":164,"prepare.run_wall_time_us":431244,"raft.queue_time_us":16,"raft.run_cpu_time_us":50,"raft.run_wall_time_us":51,"replication_time_us":675079,"thread_start_us":58,"threads_started":1}]]}
W20260808 07:01:35.804934 18477 scanner-internal.cc:458] Time spent opening tablet: real 0.892s user 0.000s sys 0.000s
W20260808 07:01:35.804941 18478 scanner-internal.cc:458] Time spent opening tablet: real 0.892s user 0.000s sys 0.000s
I20260808 07:01:36.745962 18539 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:36.746136 18539 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:36.746402 18539 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:36.746565 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:36.746625 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:36.746901 18041 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:36.747015 18539 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:36.747077 18560 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=18, attempt_no=0}) took 1984 ms. Trace:
I20260808 07:01:36.747136 18560 rpcz_store.cc:276] 0808 07:01:34.763069 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:34.763132 (+ 63us) service_pool.cc:227] Handling call
0808 07:01:36.747068 (+1983936us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:35.129266 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:35.129279 (+ 13us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:35.129279 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:01:35.129280 (+ 1us) tablet.cc:660] Decoding operations
0808 07:01:35.129290 (+ 10us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:35.129292 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:35.129293 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:35.129303 (+ 10us) tablet.cc:699] Row locks acquired
0808 07:01:35.129304 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:01:35.129307 (+ 3us) write_op.cc:270] Start()
0808 07:01:35.490470 (+361163us) write_op.cc:276] Timestamp: P: 1786172495129250 usec, L: 0
0808 07:01:35.490472 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:01:35.490538 (+ 66us) log.cc:842] Serialized 354 byte log entry
0808 07:01:36.746889 (+1256351us) op_driver.cc:444] REPLICATION: finished
0808 07:01:36.746967 (+ 78us) write_op.cc:301] APPLY: starting
0808 07:01:36.746972 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:36.746977 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:36.746980 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:36.746993 (+ 13us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:36.746998 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:36.746999 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:01:36.747019 (+ 20us) log.cc:842] Serialized 22 byte log entry
0808 07:01:36.747041 (+ 22us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:36.747044 (+ 3us) write_op.cc:454] Released schema lock
0808 07:01:36.747045 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":62,"num_ops":1,"prepare.queue_time_us":366008,"prepare.run_cpu_time_us":204,"prepare.run_wall_time_us":361298,"replication_time_us":1256413,"thread_start_us":45,"threads_started":1}]]}
I20260808 07:01:37.038599 18539 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:37.038738 18539 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:37.038967 18539 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:37.039158 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:37.039162 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:37.039469 18208 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60832 (request call id 42) took 1084 ms. Trace:
I20260808 07:01:37.039528 18207 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60832 (request call id 43) took 1084 ms. Trace:
I20260808 07:01:37.039626 18207 rpcz_store.cc:276] 0808 07:01:35.955506 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:35.955644 (+ 138us) service_pool.cc:227] Handling call
0808 07:01:35.955687 (+ 43us) tablet_service.cc:2780] Created scanner 986f6c5f134847dc8922961aaac1cd45 for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is f5ad8136ec674b0580a34c04c24d746b
0808 07:01:35.955714 (+ 27us) tablet_service.cc:2920] Creating iterator
0808 07:01:35.955720 (+ 6us) tablet_service.cc:3297] Waiting safe time to advance
0808 07:01:35.955721 (+ 1us) tablet_service.cc:3304] Waiting for operations to commit
0808 07:01:37.039306 (+1083585us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1083583 microseconds
0808 07:01:37.039325 (+ 19us) tablet_service.cc:2945] Iterator created
0808 07:01:37.039375 (+ 50us) tablet_service.cc:2967] Iterator init: OK
0808 07:01:37.039379 (+ 4us) tablet_service.cc:3010] has_more: true
0808 07:01:37.039385 (+ 6us) tablet_service.cc:3027] Continuing scan request
0808 07:01:37.039401 (+ 16us) tablet_service.cc:3091] Found scanner 986f6c5f134847dc8922961aaac1cd45 for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is f5ad8136ec674b0580a34c04c24d746b
0808 07:01:37.039521 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
I20260808 07:01:37.039700 18042 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:37.039542 18208 rpcz_store.cc:276] 0808 07:01:35.955485 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:35.955527 (+ 42us) service_pool.cc:227] Handling call
0808 07:01:35.955578 (+ 51us) tablet_service.cc:2780] Created scanner 0c2d7cce75f84856a8ecb99e1da11ec4 for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is 3c7bbdcde00442e0b6c150407653222f
0808 07:01:35.955618 (+ 40us) tablet_service.cc:2920] Creating iterator
0808 07:01:35.955625 (+ 7us) tablet_service.cc:3297] Waiting safe time to advance
0808 07:01:35.955627 (+ 2us) tablet_service.cc:3304] Waiting for operations to commit
0808 07:01:37.039320 (+1083693us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1083693 microseconds
0808 07:01:37.039336 (+ 16us) tablet_service.cc:2945] Iterator created
0808 07:01:37.039381 (+ 45us) tablet_service.cc:2967] Iterator init: OK
0808 07:01:37.039384 (+ 3us) tablet_service.cc:3010] has_more: true
0808 07:01:37.039390 (+ 6us) tablet_service.cc:3027] Continuing scan request
0808 07:01:37.039404 (+ 14us) tablet_service.cc:3091] Found scanner 0c2d7cce75f84856a8ecb99e1da11ec4 for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is 3c7bbdcde00442e0b6c150407653222f
0808 07:01:37.039463 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
I20260808 07:01:37.039862 18560 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=19, attempt_no=0}) took 1233 ms. Trace:
I20260808 07:01:37.039855 18539 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:37.039929 18560 rpcz_store.cc:276] 0808 07:01:35.805686 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:35.805735 (+ 49us) service_pool.cc:227] Handling call
0808 07:01:37.039471 (+1233736us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:35.805903 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:35.805915 (+ 12us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:35.805915 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:01:35.805916 (+ 1us) tablet.cc:660] Decoding operations
0808 07:01:35.805926 (+ 10us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:35.805928 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:35.805929 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:35.805942 (+ 13us) tablet.cc:699] Row locks acquired
0808 07:01:35.805942 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:01:35.805947 (+ 5us) write_op.cc:270] Start()
0808 07:01:35.951065 (+145118us) write_op.cc:276] Timestamp: P: 1786172495805871 usec, L: 0
0808 07:01:35.951067 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:01:35.951132 (+ 65us) log.cc:842] Serialized 354 byte log entry
0808 07:01:37.039087 (+1087955us) op_driver.cc:444] REPLICATION: finished
0808 07:01:37.039133 (+ 46us) write_op.cc:301] APPLY: starting
0808 07:01:37.039137 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:37.039141 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:37.039143 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:37.039204 (+ 61us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:37.039209 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:37.039209 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:01:37.039229 (+ 20us) log.cc:842] Serialized 22 byte log entry
0808 07:01:37.039443 (+ 214us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:37.039446 (+ 3us) write_op.cc:454] Released schema lock
0808 07:01:37.039447 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":31,"num_ops":1,"prepare.queue_time_us":49,"prepare.run_cpu_time_us":185,"prepare.run_wall_time_us":145261,"replication_time_us":1088015}]]}
W20260808 07:01:37.040125 18478 scanner-internal.cc:458] Time spent opening tablet: real 1.085s user 0.000s sys 0.000s
W20260808 07:01:37.040125 18477 scanner-internal.cc:458] Time spent opening tablet: real 1.085s user 0.000s sys 0.000s
I20260808 07:01:37.083370 18487 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:37.083542 18388 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.15.250.130:38799 (request call id 532) took 2524 ms. Trace:
I20260808 07:01:37.083585 18388 rpcz_store.cc:276] 0808 07:01:34.558965 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:34.558989 (+ 24us) service_pool.cc:227] Handling call
0808 07:01:34.559036 (+ 47us) raft_consensus.cc:1457] Updating replica for 3 ops
0808 07:01:34.559052 (+ 16us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 07:01:34.559052 (+ 0us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0808 07:01:34.559097 (+ 45us) write_op.cc:270] Start()
0808 07:01:35.442207 (+883110us) write_op.cc:276] Timestamp: P: 1786172493014288 usec, L: 0
0808 07:01:35.442395 (+ 188us) write_op.cc:270] Start()
0808 07:01:36.087519 (+645124us) write_op.cc:276] Timestamp: P: 1786172493986574 usec, L: 0
0808 07:01:36.087730 (+ 211us) write_op.cc:270] Start()
0808 07:01:37.082841 (+995111us) write_op.cc:276] Timestamp: P: 1786172494313414 usec, L: 0
0808 07:01:37.083030 (+ 189us) log.cc:842] Serialized 1062 byte log entry
0808 07:01:37.083105 (+ 75us) raft_consensus.cc:1631] Marking committed up to 16
0808 07:01:37.083201 (+ 96us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 07:01:37.083214 (+ 13us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 07:01:37.083527 (+ 313us) raft_consensus.cc:1671] finished
0808 07:01:37.083528 (+ 1us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 07:01:37.083538 (+ 10us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:35.442419 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:35.442434 (+ 15us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:35.442435 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:01:35.442435 (+ 0us) tablet.cc:660] Decoding operations
0808 07:01:35.442448 (+ 13us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:35.442452 (+ 4us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:35.442453 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:35.442462 (+ 9us) tablet.cc:699] Row locks acquired
0808 07:01:35.442463 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:01:37.083112 (+1640649us) op_driver.cc:444] REPLICATION: finished
0808 07:01:37.083246 (+ 134us) write_op.cc:301] APPLY: starting
0808 07:01:37.083250 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:37.083255 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:37.083256 (+ 1us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:37.083285 (+ 29us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:37.083291 (+ 6us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:37.083292 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:01:37.083312 (+ 20us) log.cc:842] Serialized 22 byte log entry
0808 07:01:37.083319 (+ 7us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:37.083322 (+ 3us) write_op.cc:454] Released schema lock
0808 07:01:37.083323 (+ 1us) write_op.cc:341] FINISH: Updating metrics
0808 07:01:37.083338 (+ 15us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:37.083339 (+ 1us) write_op.cc:454] Released schema lock
Related trace 'op':
0808 07:01:36.087785 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:36.087800 (+ 15us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:36.087801 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:01:36.087801 (+ 0us) tablet.cc:660] Decoding operations
0808 07:01:36.087814 (+ 13us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:36.087818 (+ 4us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:36.087819 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:36.087828 (+ 9us) tablet.cc:699] Row locks acquired
0808 07:01:36.087828 (+ 0us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0808 07:01:37.083042 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:37.083057 (+ 15us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:37.083057 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:01:37.083058 (+ 1us) tablet.cc:660] Decoding operations
0808 07:01:37.083071 (+ 13us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:37.083075 (+ 4us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:37.083076 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:37.083085 (+ 9us) tablet.cc:699] Row locks acquired
0808 07:01:37.083086 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":66,"threads_started":1,"wal-append.queue_time_us":90,"child_traces":[["op",{"apply.queue_time_us":115,"apply.run_cpu_time_us":115,"apply.run_wall_time_us":115,"num_ops":1,"partition_lock_wait_us":2,"prepare.queue_time_us":161,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":69,"replication_time_us":2524011,"thread_start_us":193,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":1,"prepare.queue_time_us":209,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":70,"thread_start_us":124,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":1,"prepare.queue_time_us":153,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":60,"thread_start_us":102,"threads_started":1}]]}
I20260808 07:01:38.001549 18487 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:38.001659 18487 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:38.001855 18487 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 910bd273924d4cb0adc0f7b65e454f19 (127.15.250.129:45605), 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273)
I20260808 07:01:38.002146 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:38.002197 18111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "910bd273924d4cb0adc0f7b65e454f19" is_pre_election: true
I20260808 07:01:38.002493 18322 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 221e532665ab4c7f9585be8ed849b6a9; no voters: 03f850c83dff43fdade5214b4dbee7a1, 910bd273924d4cb0adc0f7b65e454f19
I20260808 07:01:38.006157 18539 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:38.006268 18539 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:38.006490 18539 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:38.006646 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:38.006839 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:38.007246 18041 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:38.007360 18539 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 07:01:38.007439 18477 scanner-internal.cc:458] Time spent opening tablet: real 0.817s user 0.000s sys 0.000s
I20260808 07:01:38.007459 18575 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=20, attempt_no=0}) took 1259 ms. Trace:
W20260808 07:01:38.007601 18478 scanner-internal.cc:458] Time spent opening tablet: real 0.817s user 0.000s sys 0.000s
I20260808 07:01:38.007524 18575 rpcz_store.cc:276] 0808 07:01:36.747711 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:36.747736 (+ 25us) service_pool.cc:227] Handling call
0808 07:01:38.007451 (+1259715us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:36.747952 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:36.747964 (+ 12us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:36.747964 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:01:36.747965 (+ 1us) tablet.cc:660] Decoding operations
0808 07:01:36.747974 (+ 9us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:36.747975 (+ 1us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:36.747976 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:36.747983 (+ 7us) tablet.cc:699] Row locks acquired
0808 07:01:36.747983 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:01:36.747990 (+ 7us) write_op.cc:270] Start()
0808 07:01:37.478165 (+730175us) write_op.cc:276] Timestamp: P: 1786172496747938 usec, L: 0
0808 07:01:37.478167 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:01:37.478300 (+ 133us) log.cc:842] Serialized 354 byte log entry
0808 07:01:38.006673 (+528373us) op_driver.cc:444] REPLICATION: finished
0808 07:01:38.006861 (+ 188us) write_op.cc:301] APPLY: starting
0808 07:01:38.006865 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:38.006870 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:38.006873 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:38.006885 (+ 12us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:38.006891 (+ 6us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:38.006891 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:01:38.006922 (+ 31us) log.cc:842] Serialized 22 byte log entry
0808 07:01:38.007428 (+ 506us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:38.007431 (+ 3us) write_op.cc:454] Released schema lock
0808 07:01:38.007432 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":174,"num_ops":1,"prepare.queue_time_us":98,"prepare.run_cpu_time_us":308,"prepare.run_wall_time_us":730398,"raft.queue_time_us":15,"raft.run_cpu_time_us":76,"raft.run_wall_time_us":76,"replication_time_us":528502,"thread_start_us":131,"threads_started":2}]]}
I20260808 07:01:38.705890 18487 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:38.710785 18539 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:38.710899 18539 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:38.711153 18539 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:38.711290 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:38.711362 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:38.711639 18041 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:38.711689 18584 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=21, attempt_no=0}) took 1671 ms. Trace:
I20260808 07:01:38.711742 18584 rpcz_store.cc:276] 0808 07:01:37.040620 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:37.040670 (+ 50us) service_pool.cc:227] Handling call
0808 07:01:38.711680 (+1671010us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:37.478366 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:37.478383 (+ 17us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:37.478383 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:01:37.478384 (+ 1us) tablet.cc:660] Decoding operations
0808 07:01:37.478398 (+ 14us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:37.478401 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:37.478402 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:37.478413 (+ 11us) tablet.cc:699] Row locks acquired
0808 07:01:37.478413 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:01:37.478418 (+ 5us) write_op.cc:270] Start()
0808 07:01:38.064543 (+586125us) write_op.cc:276] Timestamp: P: 1786172497478399 usec, L: 0
0808 07:01:38.064545 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:01:38.064619 (+ 74us) log.cc:842] Serialized 354 byte log entry
0808 07:01:38.711413 (+646794us) op_driver.cc:444] REPLICATION: finished
0808 07:01:38.711589 (+ 176us) write_op.cc:301] APPLY: starting
0808 07:01:38.711595 (+ 6us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:38.711599 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:38.711602 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:38.711615 (+ 13us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:38.711620 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:38.711621 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:01:38.711641 (+ 20us) log.cc:842] Serialized 22 byte log entry
0808 07:01:38.711657 (+ 16us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:38.711661 (+ 4us) write_op.cc:454] Released schema lock
0808 07:01:38.711661 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":163,"num_ops":1,"prepare.queue_time_us":437576,"prepare.run_cpu_time_us":244,"prepare.run_wall_time_us":586329,"raft.queue_time_us":28,"raft.run_cpu_time_us":69,"raft.run_wall_time_us":69,"replication_time_us":646863,"thread_start_us":46,"threads_started":1}]]}
I20260808 07:01:39.215261 18539 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:39.216148 18589 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=22, attempt_no=0}) took 1208 ms. Trace:
I20260808 07:01:39.216380 18208 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60832 (request call id 49) took 1058 ms. Trace:
I20260808 07:01:39.216344 18207 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60832 (request call id 50) took 1058 ms. Trace:
I20260808 07:01:39.216435 18208 rpcz_store.cc:276] 0808 07:01:38.158188 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:38.158243 (+ 55us) service_pool.cc:227] Handling call
0808 07:01:38.158309 (+ 66us) tablet_service.cc:2780] Created scanner 8fc07dbf87dc4520b7cf1773d1de0e5e for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is 1bb90f75b8ef4d718eb297105ebf8849
0808 07:01:38.158348 (+ 39us) tablet_service.cc:2920] Creating iterator
0808 07:01:38.158398 (+ 50us) tablet_service.cc:3297] Waiting safe time to advance
0808 07:01:38.344052 (+185654us) tablet_service.cc:3304] Waiting for operations to commit
0808 07:01:39.216155 (+872103us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1057754 microseconds
0808 07:01:39.216169 (+ 14us) tablet_service.cc:2945] Iterator created
0808 07:01:39.216215 (+ 46us) tablet_service.cc:2967] Iterator init: OK
0808 07:01:39.216218 (+ 3us) tablet_service.cc:3010] has_more: true
0808 07:01:39.216223 (+ 5us) tablet_service.cc:3027] Continuing scan request
0808 07:01:39.216238 (+ 15us) tablet_service.cc:3091] Found scanner 8fc07dbf87dc4520b7cf1773d1de0e5e for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is 1bb90f75b8ef4d718eb297105ebf8849
0808 07:01:39.216377 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":13440}
I20260808 07:01:39.216308 18589 rpcz_store.cc:276] 0808 07:01:38.008076 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:38.008125 (+ 49us) service_pool.cc:227] Handling call
0808 07:01:39.216135 (+1208010us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:38.064705 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:38.064722 (+ 17us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:38.064722 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:01:38.064723 (+ 1us) tablet.cc:660] Decoding operations
0808 07:01:38.064735 (+ 12us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:38.064738 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:38.064739 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:38.064751 (+ 12us) tablet.cc:699] Row locks acquired
0808 07:01:38.064752 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:01:38.064760 (+ 8us) write_op.cc:270] Start()
0808 07:01:38.343910 (+279150us) write_op.cc:276] Timestamp: P: 1786172498064725 usec, L: 0
0808 07:01:38.343912 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:01:38.344064 (+ 152us) log.cc:842] Serialized 354 byte log entry
0808 07:01:39.215876 (+871812us) op_driver.cc:444] REPLICATION: finished
0808 07:01:39.215993 (+ 117us) write_op.cc:301] APPLY: starting
0808 07:01:39.215997 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:39.216000 (+ 3us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:39.216003 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:39.216016 (+ 13us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:39.216020 (+ 4us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:39.216020 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:01:39.216038 (+ 18us) log.cc:842] Serialized 22 byte log entry
0808 07:01:39.216107 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:39.216110 (+ 3us) write_op.cc:454] Released schema lock
0808 07:01:39.216111 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":108,"num_ops":1,"prepare.queue_time_us":56458,"prepare.run_cpu_time_us":298,"prepare.run_wall_time_us":279386,"replication_time_us":871960,"thread_start_us":69,"threads_started":1}]]}
I20260808 07:01:39.216470 18207 rpcz_store.cc:276] 0808 07:01:38.158226 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:38.158277 (+ 51us) service_pool.cc:227] Handling call
0808 07:01:38.158342 (+ 65us) tablet_service.cc:2780] Created scanner 92f5c9f8f3824121b69bb87697f17ae4 for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is 8324d2118ea7486890bf0fe6a8ce5749
0808 07:01:38.158381 (+ 39us) tablet_service.cc:2920] Creating iterator
0808 07:01:38.158398 (+ 17us) tablet_service.cc:3297] Waiting safe time to advance
0808 07:01:38.344016 (+185618us) tablet_service.cc:3304] Waiting for operations to commit
0808 07:01:39.216115 (+872099us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1057712 microseconds
0808 07:01:39.216136 (+ 21us) tablet_service.cc:2945] Iterator created
0808 07:01:39.216224 (+ 88us) tablet_service.cc:2967] Iterator init: OK
0808 07:01:39.216228 (+ 4us) tablet_service.cc:3010] has_more: true
0808 07:01:39.216234 (+ 6us) tablet_service.cc:3027] Continuing scan request
0808 07:01:39.216246 (+ 12us) tablet_service.cc:3091] Found scanner 92f5c9f8f3824121b69bb87697f17ae4 for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is 8324d2118ea7486890bf0fe6a8ce5749
0808 07:01:39.216337 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":8064}
W20260808 07:01:39.216920 18478 scanner-internal.cc:458] Time spent opening tablet: real 1.059s user 0.000s sys 0.000s
W20260808 07:01:39.217008 18477 scanner-internal.cc:458] Time spent opening tablet: real 1.059s user 0.000s sys 0.000s
I20260808 07:01:39.788755 18487 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:39.788765 18388 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.15.250.130:38799 (request call id 535) took 1082 ms. Trace:
I20260808 07:01:39.788916 18487 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:39.788916 18388 rpcz_store.cc:276] 0808 07:01:38.706619 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:38.706664 (+ 45us) service_pool.cc:227] Handling call
0808 07:01:38.706709 (+ 45us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 07:01:38.706733 (+ 24us) raft_consensus.cc:1520] Early marking committed up to index 22
0808 07:01:38.706809 (+ 76us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 07:01:38.706855 (+ 46us) write_op.cc:270] Start()
0808 07:01:39.307983 (+601128us) write_op.cc:276] Timestamp: P: 1786172497478399 usec, L: 0
0808 07:01:39.308149 (+ 166us) write_op.cc:270] Start()
0808 07:01:39.788326 (+480177us) write_op.cc:276] Timestamp: P: 1786172498064725 usec, L: 0
0808 07:01:39.788428 (+ 102us) log.cc:842] Serialized 708 byte log entry
0808 07:01:39.788511 (+ 83us) raft_consensus.cc:1631] Marking committed up to 22
0808 07:01:39.788512 (+ 1us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 07:01:39.788737 (+ 225us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 07:01:39.788740 (+ 3us) raft_consensus.cc:1671] finished
0808 07:01:39.788740 (+ 0us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 07:01:39.788759 (+ 19us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:39.308185 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:39.308206 (+ 21us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:39.308207 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:01:39.308208 (+ 1us) tablet.cc:660] Decoding operations
0808 07:01:39.308225 (+ 17us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:39.308230 (+ 5us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:39.308230 (+ 0us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:39.308241 (+ 11us) tablet.cc:699] Row locks acquired
0808 07:01:39.308241 (+ 0us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0808 07:01:39.788454 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:39.788469 (+ 15us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:39.788469 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:01:39.788470 (+ 1us) tablet.cc:660] Decoding operations
0808 07:01:39.788484 (+ 14us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:39.788488 (+ 4us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:39.788489 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:39.788522 (+ 33us) tablet.cc:699] Row locks acquired
0808 07:01:39.788523 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":73,"threads_started":1,"wal-append.queue_time_us":126,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":2,"prepare.queue_time_us":147,"prepare.run_cpu_time_us":107,"prepare.run_wall_time_us":108,"thread_start_us":91,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":2,"prepare.queue_time_us":56,"prepare.run_cpu_time_us":117,"prepare.run_wall_time_us":118}]]}
I20260808 07:01:39.789145 18487 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 910bd273924d4cb0adc0f7b65e454f19 (127.15.250.129:45605), 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273)
I20260808 07:01:39.789462 18111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "910bd273924d4cb0adc0f7b65e454f19" is_pre_election: true
I20260808 07:01:39.789656 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:39.789888 18322 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 221e532665ab4c7f9585be8ed849b6a9; no voters: 03f850c83dff43fdade5214b4dbee7a1, 910bd273924d4cb0adc0f7b65e454f19
I20260808 07:01:39.893172 18487 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:39.894086 18597 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=23, attempt_no=0}) took 1181 ms. Trace:
I20260808 07:01:39.894212 18597 rpcz_store.cc:276] 0808 07:01:38.712383 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:38.712435 (+ 52us) service_pool.cc:227] Handling call
0808 07:01:39.894073 (+1181638us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:38.712635 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:38.712646 (+ 11us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:38.712646 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:01:38.712646 (+ 0us) tablet.cc:660] Decoding operations
0808 07:01:38.712656 (+ 10us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:38.712658 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:38.712659 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:38.712668 (+ 9us) tablet.cc:699] Row locks acquired
0808 07:01:38.712669 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:01:38.712676 (+ 7us) write_op.cc:270] Start()
0808 07:01:39.175808 (+463132us) write_op.cc:276] Timestamp: P: 1786172498712643 usec, L: 0
0808 07:01:39.175810 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:01:39.175877 (+ 67us) log.cc:842] Serialized 354 byte log entry
0808 07:01:39.893861 (+717984us) op_driver.cc:444] REPLICATION: finished
0808 07:01:39.893963 (+ 102us) write_op.cc:301] APPLY: starting
0808 07:01:39.893967 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:39.893971 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:39.893974 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:39.893987 (+ 13us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:39.893991 (+ 4us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:39.893992 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:01:39.894009 (+ 17us) log.cc:842] Serialized 22 byte log entry
0808 07:01:39.894046 (+ 37us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:39.894049 (+ 3us) write_op.cc:454] Released schema lock
0808 07:01:39.894050 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":94,"num_ops":1,"prepare.queue_time_us":63,"prepare.run_cpu_time_us":193,"prepare.run_wall_time_us":463273,"replication_time_us":718046,"thread_start_us":54,"threads_started":1}]]}
I20260808 07:01:39.969643 18539 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:39.969789 18539 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:39.970034 18539 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:39.970139 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:39.970338 18388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:39.970587 18041 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:39.970758 18539 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:40.682320 18487 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:40.682480 18487 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:40.682683 18487 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 910bd273924d4cb0adc0f7b65e454f19 (127.15.250.129:45605), 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273)
I20260808 07:01:40.682971 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:40.682945 18111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "910bd273924d4cb0adc0f7b65e454f19" is_pre_election: true
I20260808 07:01:40.683434 18539 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:40.683494 18539 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:40.683590 18208 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60832 (request call id 54) took 1316 ms. Trace:
I20260808 07:01:40.683683 18539 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:40.683650 18208 rpcz_store.cc:276] 0808 07:01:39.367566 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:39.367619 (+ 53us) service_pool.cc:227] Handling call
0808 07:01:39.367677 (+ 58us) tablet_service.cc:2780] Created scanner 858b9dcd02084a70930136d5b3363638 for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is fcf1947e00064ede90536e01e01921e9
0808 07:01:39.367707 (+ 30us) tablet_service.cc:2920] Creating iterator
0808 07:01:39.367717 (+ 10us) tablet_service.cc:3297] Waiting safe time to advance
0808 07:01:40.171584 (+803867us) tablet_service.cc:3304] Waiting for operations to commit
0808 07:01:40.683432 (+511848us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1315711 microseconds
0808 07:01:40.683449 (+ 17us) tablet_service.cc:2945] Iterator created
0808 07:01:40.683495 (+ 46us) tablet_service.cc:2967] Iterator init: OK
0808 07:01:40.683498 (+ 3us) tablet_service.cc:3010] has_more: true
0808 07:01:40.683503 (+ 5us) tablet_service.cc:3027] Continuing scan request
0808 07:01:40.683518 (+ 15us) tablet_service.cc:3091] Found scanner 858b9dcd02084a70930136d5b3363638 for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is fcf1947e00064ede90536e01e01921e9
0808 07:01:40.683582 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":22528}
I20260808 07:01:40.683815 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:40.683913 18207 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60832 (request call id 53) took 1316 ms. Trace:
I20260808 07:01:40.683799 18603 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=24, attempt_no=0}) took 1466 ms. Trace:
W20260808 07:01:40.684003 18477 scanner-internal.cc:458] Time spent opening tablet: real 1.317s user 0.000s sys 0.000s
I20260808 07:01:40.683991 18603 rpcz_store.cc:276] 0808 07:01:39.217157 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:39.217238 (+ 81us) service_pool.cc:227] Handling call
0808 07:01:40.683793 (+1466555us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:39.217414 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:39.217426 (+ 12us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:39.217427 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:01:39.217427 (+ 0us) tablet.cc:660] Decoding operations
0808 07:01:39.217436 (+ 9us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:39.217438 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:39.217439 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:39.217447 (+ 8us) tablet.cc:699] Row locks acquired
0808 07:01:39.217447 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:01:39.217451 (+ 4us) write_op.cc:270] Start()
0808 07:01:40.171537 (+954086us) write_op.cc:276] Timestamp: P: 1786172499217397 usec, L: 0
0808 07:01:40.171538 (+ 1us) op_driver.cc:331] REPLICATION: starting
0808 07:01:40.171629 (+ 91us) log.cc:842] Serialized 354 byte log entry
0808 07:01:40.683046 (+511417us) op_driver.cc:444] REPLICATION: finished
0808 07:01:40.683290 (+ 244us) write_op.cc:301] APPLY: starting
0808 07:01:40.683294 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:40.683300 (+ 6us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:40.683302 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:40.683315 (+ 13us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:40.683321 (+ 6us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:40.683322 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:01:40.683339 (+ 17us) log.cc:842] Serialized 22 byte log entry
0808 07:01:40.683768 (+ 429us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:40.683772 (+ 4us) write_op.cc:454] Released schema lock
0808 07:01:40.683772 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":226,"num_ops":1,"prepare.queue_time_us":30,"prepare.run_cpu_time_us":210,"prepare.run_wall_time_us":954266,"raft.queue_time_us":51,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":85,"replication_time_us":511503,"spinlock_wait_cycles":13952,"thread_start_us":176,"threads_started":1}]]}
I20260808 07:01:40.683951 18207 rpcz_store.cc:276] 0808 07:01:39.367540 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:39.367600 (+ 60us) service_pool.cc:227] Handling call
0808 07:01:39.367663 (+ 63us) tablet_service.cc:2780] Created scanner b6ee63bdc5d54c7cb3fa3c00a23a6f55 for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is 6126ef6e07d743e584248277b4f7a62a
0808 07:01:39.367705 (+ 42us) tablet_service.cc:2920] Creating iterator
0808 07:01:39.367723 (+ 18us) tablet_service.cc:3297] Waiting safe time to advance
0808 07:01:40.171609 (+803886us) tablet_service.cc:3304] Waiting for operations to commit
0808 07:01:40.683792 (+512183us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1316065 microseconds
0808 07:01:40.683809 (+ 17us) tablet_service.cc:2945] Iterator created
0808 07:01:40.683843 (+ 34us) tablet_service.cc:2967] Iterator init: OK
0808 07:01:40.683846 (+ 3us) tablet_service.cc:3010] has_more: true
0808 07:01:40.683851 (+ 5us) tablet_service.cc:3027] Continuing scan request
0808 07:01:40.683862 (+ 11us) tablet_service.cc:3091] Found scanner b6ee63bdc5d54c7cb3fa3c00a23a6f55 for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is 6126ef6e07d743e584248277b4f7a62a
0808 07:01:40.683910 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
I20260808 07:01:40.683615 18322 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 221e532665ab4c7f9585be8ed849b6a9; no voters: 03f850c83dff43fdade5214b4dbee7a1, 910bd273924d4cb0adc0f7b65e454f19
W20260808 07:01:40.684427 18478 scanner-internal.cc:458] Time spent opening tablet: real 1.317s user 0.000s sys 0.000s
I20260808 07:01:40.684552 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:40.684890 18041 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:40.927484 18539 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:40.928344 18603 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=25, attempt_no=0}) took 1033 ms. Trace:
I20260808 07:01:40.928428 18603 rpcz_store.cc:276] 0808 07:01:39.894877 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:39.894914 (+ 37us) service_pool.cc:227] Handling call
0808 07:01:40.928334 (+1033420us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:40.171688 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:40.171705 (+ 17us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:40.171706 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:01:40.171706 (+ 0us) tablet.cc:660] Decoding operations
0808 07:01:40.171720 (+ 14us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:40.171723 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:40.171724 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:40.171737 (+ 13us) tablet.cc:699] Row locks acquired
0808 07:01:40.171737 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:01:40.171742 (+ 5us) write_op.cc:270] Start()
0808 07:01:40.325865 (+154123us) write_op.cc:276] Timestamp: P: 1786172500171699 usec, L: 0
0808 07:01:40.325868 (+ 3us) op_driver.cc:331] REPLICATION: starting
0808 07:01:40.325933 (+ 65us) log.cc:842] Serialized 354 byte log entry
0808 07:01:40.927959 (+602026us) op_driver.cc:444] REPLICATION: finished
0808 07:01:40.928193 (+ 234us) write_op.cc:301] APPLY: starting
0808 07:01:40.928197 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:40.928200 (+ 3us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:40.928202 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:40.928243 (+ 41us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:40.928249 (+ 6us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:40.928249 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:01:40.928291 (+ 42us) log.cc:842] Serialized 22 byte log entry
0808 07:01:40.928309 (+ 18us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:40.928312 (+ 3us) write_op.cc:454] Released schema lock
0808 07:01:40.928312 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":48,"num_ops":1,"prepare.queue_time_us":276682,"prepare.run_cpu_time_us":219,"prepare.run_wall_time_us":154294,"replication_time_us":602086}]]}
I20260808 07:01:41.222761 18487 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:41.459658 18539 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:41.459793 18539 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:41.460021 18539 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:41.460283 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:41.460299 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:41.460569 18042 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:41.460729 18539 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 07:01:41.461333 18477 scanner-internal.cc:458] Time spent opening tablet: real 0.627s user 0.000s sys 0.000s
W20260808 07:01:41.461418 18478 scanner-internal.cc:458] Time spent opening tablet: real 0.627s user 0.000s sys 0.000s
I20260808 07:01:41.669909 18487 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:41.670024 18487 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:41.670279 18487 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 910bd273924d4cb0adc0f7b65e454f19 (127.15.250.129:45605), 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273)
I20260808 07:01:41.670429 18112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "910bd273924d4cb0adc0f7b65e454f19" is_pre_election: true
I20260808 07:01:41.670526 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:41.670739 18322 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 221e532665ab4c7f9585be8ed849b6a9; no voters: 03f850c83dff43fdade5214b4dbee7a1, 910bd273924d4cb0adc0f7b65e454f19
I20260808 07:01:41.670900 18487 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:42.495213 18487 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:42.495328 18487 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:42.495548 18487 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 910bd273924d4cb0adc0f7b65e454f19 (127.15.250.129:45605), 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273)
I20260808 07:01:42.495769 18112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "910bd273924d4cb0adc0f7b65e454f19" is_pre_election: true
I20260808 07:01:42.495759 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:42.496055 18322 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 221e532665ab4c7f9585be8ed849b6a9; no voters: 03f850c83dff43fdade5214b4dbee7a1, 910bd273924d4cb0adc0f7b65e454f19
I20260808 07:01:42.496169 18487 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:42.496307 18619 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=27, attempt_no=0}) took 1567 ms. Trace:
I20260808 07:01:42.496481 18619 rpcz_store.cc:276] 0808 07:01:40.929017 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:40.929058 (+ 41us) service_pool.cc:227] Handling call
0808 07:01:42.496299 (+1567241us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:40.929226 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:40.929237 (+ 11us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:40.929237 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:01:40.929237 (+ 0us) tablet.cc:660] Decoding operations
0808 07:01:40.929247 (+ 10us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:40.929248 (+ 1us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:40.929249 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:40.929256 (+ 7us) tablet.cc:699] Row locks acquired
0808 07:01:40.929257 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:01:40.929263 (+ 6us) write_op.cc:270] Start()
0808 07:01:41.862327 (+933064us) write_op.cc:276] Timestamp: P: 1786172500929209 usec, L: 0
0808 07:01:41.862328 (+ 1us) op_driver.cc:331] REPLICATION: starting
0808 07:01:41.862429 (+ 101us) log.cc:842] Serialized 354 byte log entry
0808 07:01:42.496062 (+633633us) op_driver.cc:444] REPLICATION: finished
0808 07:01:42.496164 (+ 102us) write_op.cc:301] APPLY: starting
0808 07:01:42.496168 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:42.496173 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:42.496175 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:42.496189 (+ 14us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:42.496195 (+ 6us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:42.496196 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:01:42.496216 (+ 20us) log.cc:842] Serialized 22 byte log entry
0808 07:01:42.496273 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:42.496276 (+ 3us) write_op.cc:454] Released schema lock
0808 07:01:42.496277 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":87,"num_ops":1,"prepare.queue_time_us":37,"prepare.run_cpu_time_us":232,"prepare.run_wall_time_us":933253,"raft.queue_time_us":46,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":105,"replication_time_us":633729,"spinlock_wait_cycles":3328,"thread_start_us":60,"threads_started":1}]]}
W20260808 07:01:42.496630 18477 scanner-internal.cc:458] Time spent opening tablet: real 0.885s user 0.000s sys 0.000s
W20260808 07:01:42.496680 18478 scanner-internal.cc:458] Time spent opening tablet: real 0.885s user 0.000s sys 0.000s
I20260808 07:01:42.773304 18539 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:42.773466 18539 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:42.773702 18539 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:42.773968 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:42.774199 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:42.774472 18041 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:43.351800 18487 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:43.351931 18487 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:43.352175 18487 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 910bd273924d4cb0adc0f7b65e454f19 (127.15.250.129:45605), 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273)
I20260808 07:01:43.352428 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:43.352453 18112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "910bd273924d4cb0adc0f7b65e454f19" is_pre_election: true
I20260808 07:01:43.352754 18627 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=28, attempt_no=0}) took 1891 ms. Trace:
I20260808 07:01:43.352782 18322 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 221e532665ab4c7f9585be8ed849b6a9; no voters: 03f850c83dff43fdade5214b4dbee7a1, 910bd273924d4cb0adc0f7b65e454f19
I20260808 07:01:43.352862 18487 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:43.352818 18627 rpcz_store.cc:276] 0808 07:01:41.461493 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:41.461531 (+ 38us) service_pool.cc:227] Handling call
0808 07:01:43.352747 (+1891216us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:41.862488 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:41.862504 (+ 16us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:41.862504 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:01:41.862505 (+ 1us) tablet.cc:660] Decoding operations
0808 07:01:41.862517 (+ 12us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:41.862520 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:41.862522 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:41.862533 (+ 11us) tablet.cc:699] Row locks acquired
0808 07:01:41.862534 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:01:41.862539 (+ 5us) write_op.cc:270] Start()
0808 07:01:42.157678 (+295139us) write_op.cc:276] Timestamp: P: 1786172501862490 usec, L: 0
0808 07:01:42.157681 (+ 3us) op_driver.cc:331] REPLICATION: starting
0808 07:01:42.157749 (+ 68us) log.cc:842] Serialized 354 byte log entry
0808 07:01:43.352494 (+1194745us) op_driver.cc:444] REPLICATION: finished
0808 07:01:43.352595 (+ 101us) write_op.cc:301] APPLY: starting
0808 07:01:43.352600 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:43.352605 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:43.352608 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:43.352620 (+ 12us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:43.352626 (+ 6us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:43.352627 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:01:43.352678 (+ 51us) log.cc:842] Serialized 22 byte log entry
0808 07:01:43.352723 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:43.352727 (+ 4us) write_op.cc:454] Released schema lock
0808 07:01:43.352727 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":87,"num_ops":1,"prepare.queue_time_us":400842,"prepare.run_cpu_time_us":201,"prepare.run_wall_time_us":295295,"replication_time_us":1194807,"thread_start_us":51,"threads_started":1}]]}
I20260808 07:01:43.363582 18539 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:43.593791 18539 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:43.593945 18539 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:43.594206 18539 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:43.594403 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:43.594527 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:43.594693 18627 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=29, attempt_no=0}) took 1097 ms. Trace:
I20260808 07:01:43.594817 18627 rpcz_store.cc:276] 0808 07:01:42.497260 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:42.497319 (+ 59us) service_pool.cc:227] Handling call
0808 07:01:43.594686 (+1097367us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:42.497455 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:42.497464 (+ 9us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:42.497465 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:01:42.497465 (+ 0us) tablet.cc:660] Decoding operations
0808 07:01:42.497473 (+ 8us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:42.497475 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:42.497475 (+ 0us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:42.497481 (+ 6us) tablet.cc:699] Row locks acquired
0808 07:01:42.497482 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:01:42.497487 (+ 5us) write_op.cc:270] Start()
0808 07:01:43.026592 (+529105us) write_op.cc:276] Timestamp: P: 1786172502497471 usec, L: 0
0808 07:01:43.026595 (+ 3us) op_driver.cc:331] REPLICATION: starting
0808 07:01:43.026728 (+ 133us) log.cc:842] Serialized 354 byte log entry
0808 07:01:43.594360 (+567632us) op_driver.cc:444] REPLICATION: finished
0808 07:01:43.594415 (+ 55us) write_op.cc:301] APPLY: starting
0808 07:01:43.594419 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:43.594423 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:43.594425 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:43.594439 (+ 14us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:43.594445 (+ 6us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:43.594446 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:01:43.594467 (+ 21us) log.cc:842] Serialized 22 byte log entry
0808 07:01:43.594661 (+ 194us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:43.594665 (+ 4us) write_op.cc:454] Released schema lock
0808 07:01:43.594665 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":39,"num_ops":1,"prepare.queue_time_us":40,"prepare.run_cpu_time_us":239,"prepare.run_wall_time_us":529307,"replication_time_us":567761}]]}
W20260808 07:01:43.594940 18478 scanner-internal.cc:458] Time spent opening tablet: real 0.948s user 0.000s sys 0.000s
W20260808 07:01:43.595044 18477 scanner-internal.cc:458] Time spent opening tablet: real 0.948s user 0.000s sys 0.000s
I20260808 07:01:43.595266 18041 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:43.595391 18539 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:44.053643 18487 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:44.053761 18487 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:44.053978 18487 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 910bd273924d4cb0adc0f7b65e454f19 (127.15.250.129:45605), 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273)
I20260808 07:01:44.054293 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:44.054273 18112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "910bd273924d4cb0adc0f7b65e454f19" is_pre_election: true
I20260808 07:01:44.054576 18322 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 221e532665ab4c7f9585be8ed849b6a9; no voters: 03f850c83dff43fdade5214b4dbee7a1, 910bd273924d4cb0adc0f7b65e454f19
I20260808 07:01:44.511955 18539 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:44.512081 18539 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:44.512338 18539 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:44.512549 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:44.512539 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:44.512754 18635 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=30, attempt_no=0}) took 1159 ms. Trace:
I20260808 07:01:44.512849 18635 rpcz_store.cc:276] 0808 07:01:43.353533 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:43.353576 (+ 43us) service_pool.cc:227] Handling call
0808 07:01:44.512744 (+1159168us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:43.353726 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:43.353736 (+ 10us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:43.353737 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:01:43.353737 (+ 0us) tablet.cc:660] Decoding operations
0808 07:01:43.353748 (+ 11us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:43.353749 (+ 1us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:43.353750 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:43.353758 (+ 8us) tablet.cc:699] Row locks acquired
0808 07:01:43.353758 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:01:43.353781 (+ 23us) write_op.cc:270] Start()
0808 07:01:43.997851 (+644070us) write_op.cc:276] Timestamp: P: 1786172503353715 usec, L: 0
0808 07:01:43.997853 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:01:43.997940 (+ 87us) log.cc:842] Serialized 354 byte log entry
0808 07:01:44.512526 (+514586us) op_driver.cc:444] REPLICATION: finished
0808 07:01:44.512617 (+ 91us) write_op.cc:301] APPLY: starting
0808 07:01:44.512620 (+ 3us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:44.512624 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:44.512627 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:44.512659 (+ 32us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:44.512664 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:44.512665 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:01:44.512685 (+ 20us) log.cc:842] Serialized 22 byte log entry
0808 07:01:44.512718 (+ 33us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:44.512722 (+ 4us) write_op.cc:454] Released schema lock
0808 07:01:44.512722 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":81,"num_ops":1,"prepare.queue_time_us":37,"prepare.run_cpu_time_us":198,"prepare.run_wall_time_us":644237,"replication_time_us":514670,"thread_start_us":46,"threads_started":1}]]}
W20260808 07:01:44.513176 18478 scanner-internal.cc:458] Time spent opening tablet: real 0.768s user 0.000s sys 0.000s
W20260808 07:01:44.513288 18477 scanner-internal.cc:458] Time spent opening tablet: real 0.768s user 0.000s sys 0.000s
I20260808 07:01:44.513273 18042 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:45.019002 18487 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:45.145715 18539 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:45.146610 18643 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=31, attempt_no=0}) took 1551 ms. Trace:
I20260808 07:01:45.146709 18643 rpcz_store.cc:276] 0808 07:01:43.595469 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:43.595508 (+ 39us) service_pool.cc:227] Handling call
0808 07:01:45.146596 (+1551088us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:43.997970 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:43.997986 (+ 16us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:43.997989 (+ 3us) write_op.cc:435] Acquired schema lock
0808 07:01:43.997989 (+ 0us) tablet.cc:660] Decoding operations
0808 07:01:43.998023 (+ 34us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:43.998027 (+ 4us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:43.998028 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:43.998041 (+ 13us) tablet.cc:699] Row locks acquired
0808 07:01:43.998041 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:01:43.998047 (+ 6us) write_op.cc:270] Start()
0808 07:01:44.327219 (+329172us) write_op.cc:276] Timestamp: P: 1786172503997993 usec, L: 0
0808 07:01:44.327221 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:01:44.327287 (+ 66us) log.cc:842] Serialized 354 byte log entry
0808 07:01:45.146316 (+819029us) op_driver.cc:444] REPLICATION: finished
0808 07:01:45.146472 (+ 156us) write_op.cc:301] APPLY: starting
0808 07:01:45.146477 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:45.146481 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:45.146484 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:45.146499 (+ 15us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:45.146504 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:45.146505 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:01:45.146548 (+ 43us) log.cc:842] Serialized 22 byte log entry
0808 07:01:45.146570 (+ 22us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:45.146573 (+ 3us) write_op.cc:454] Released schema lock
0808 07:01:45.146573 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":139,"num_ops":1,"prepare.queue_time_us":402350,"prepare.run_cpu_time_us":255,"prepare.run_wall_time_us":329341,"replication_time_us":819090,"thread_start_us":71,"threads_started":1}]]}
I20260808 07:01:45.518098 18487 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:45.518255 18487 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:45.518455 18487 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 910bd273924d4cb0adc0f7b65e454f19 (127.15.250.129:45605), 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273)
I20260808 07:01:45.518648 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:45.518688 18112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "910bd273924d4cb0adc0f7b65e454f19" is_pre_election: true
I20260808 07:01:45.519006 18322 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 221e532665ab4c7f9585be8ed849b6a9; no voters: 03f850c83dff43fdade5214b4dbee7a1, 910bd273924d4cb0adc0f7b65e454f19
I20260808 07:01:45.593611 18539 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:45.593748 18539 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:45.593984 18539 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:45.594101 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:45.594223 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:45.594359 18643 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=32, attempt_no=0}) took 1080 ms. Trace:
I20260808 07:01:45.594410 18643 rpcz_store.cc:276] 0808 07:01:44.513461 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:44.513502 (+ 41us) service_pool.cc:227] Handling call
0808 07:01:45.594353 (+1080851us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:44.513669 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:44.513678 (+ 9us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:44.513679 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:01:44.513679 (+ 0us) tablet.cc:660] Decoding operations
0808 07:01:44.513687 (+ 8us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:44.513688 (+ 1us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:44.513689 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:44.513696 (+ 7us) tablet.cc:699] Row locks acquired
0808 07:01:44.513696 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:01:44.513701 (+ 5us) write_op.cc:270] Start()
0808 07:01:45.393779 (+880078us) write_op.cc:276] Timestamp: P: 1786172504513648 usec, L: 0
0808 07:01:45.393780 (+ 1us) op_driver.cc:331] REPLICATION: starting
0808 07:01:45.393883 (+ 103us) log.cc:842] Serialized 354 byte log entry
0808 07:01:45.594206 (+200323us) op_driver.cc:444] REPLICATION: finished
0808 07:01:45.594244 (+ 38us) write_op.cc:301] APPLY: starting
0808 07:01:45.594248 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:45.594254 (+ 6us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:45.594255 (+ 1us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:45.594276 (+ 21us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:45.594282 (+ 6us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:45.594282 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:01:45.594301 (+ 19us) log.cc:842] Serialized 22 byte log entry
0808 07:01:45.594333 (+ 32us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:45.594336 (+ 3us) write_op.cc:454] Released schema lock
0808 07:01:45.594336 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":24,"num_ops":1,"prepare.queue_time_us":42,"prepare.run_cpu_time_us":188,"prepare.run_wall_time_us":880246,"replication_time_us":200422}]]}
I20260808 07:01:45.594758 18041 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
W20260808 07:01:45.594786 18477 scanner-internal.cc:458] Time spent opening tablet: real 0.931s user 0.000s sys 0.000s
W20260808 07:01:45.594861 18478 scanner-internal.cc:458] Time spent opening tablet: real 0.931s user 0.000s sys 0.000s
I20260808 07:01:45.595036 18539 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:46.122910 18539 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:46.123018 18539 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:46.123271 18539 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:46.123431 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:46.123553 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:46.123889 18041 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:46.123982 18539 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:46.151503 18487 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:47.090510 18487 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:47.090638 18487 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:47.090844 18487 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 910bd273924d4cb0adc0f7b65e454f19 (127.15.250.129:45605), 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273)
I20260808 07:01:47.091039 18112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "910bd273924d4cb0adc0f7b65e454f19" is_pre_election: true
I20260808 07:01:47.091135 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:47.091354 18322 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 221e532665ab4c7f9585be8ed849b6a9; no voters: 03f850c83dff43fdade5214b4dbee7a1, 910bd273924d4cb0adc0f7b65e454f19
I20260808 07:01:47.142388 18539 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:47.142513 18539 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:47.142720 18539 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:47.142895 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:47.142962 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:47.143380 18042 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:47.143392 18208 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60832 (request call id 75) took 1398 ms. Trace:
I20260808 07:01:47.143443 18208 rpcz_store.cc:276] 0808 07:01:45.745310 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:45.745352 (+ 42us) service_pool.cc:227] Handling call
0808 07:01:45.745405 (+ 53us) tablet_service.cc:2780] Created scanner 11fbffa22516482686ed24c6dacfbe66 for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is 629fcc1fa31a4986b76786065f68d892
0808 07:01:45.745448 (+ 43us) tablet_service.cc:2920] Creating iterator
0808 07:01:45.745460 (+ 12us) tablet_service.cc:3297] Waiting safe time to advance
0808 07:01:46.432383 (+686923us) tablet_service.cc:3304] Waiting for operations to commit
0808 07:01:47.143228 (+710845us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1397765 microseconds
0808 07:01:47.143246 (+ 18us) tablet_service.cc:2945] Iterator created
0808 07:01:47.143297 (+ 51us) tablet_service.cc:2967] Iterator init: OK
0808 07:01:47.143300 (+ 3us) tablet_service.cc:3010] has_more: true
0808 07:01:47.143306 (+ 6us) tablet_service.cc:3027] Continuing scan request
0808 07:01:47.143319 (+ 13us) tablet_service.cc:3091] Found scanner 11fbffa22516482686ed24c6dacfbe66 for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is 629fcc1fa31a4986b76786065f68d892
0808 07:01:47.143384 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":5376}
I20260808 07:01:47.143579 18659 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=34, attempt_no=0}) took 1548 ms. Trace:
I20260808 07:01:47.143695 18207 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60832 (request call id 76) took 1398 ms. Trace:
W20260808 07:01:47.143769 18477 scanner-internal.cc:458] Time spent opening tablet: real 1.399s user 0.000s sys 0.000s
I20260808 07:01:47.143759 18207 rpcz_store.cc:276] 0808 07:01:45.745331 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:45.745366 (+ 35us) service_pool.cc:227] Handling call
0808 07:01:45.745413 (+ 47us) tablet_service.cc:2780] Created scanner 7af98fc7a9a945d2afdafaf20b538ae1 for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is 416794ab884c4d79868c423114541ee2
0808 07:01:45.745448 (+ 35us) tablet_service.cc:2920] Creating iterator
0808 07:01:45.745460 (+ 12us) tablet_service.cc:3297] Waiting safe time to advance
0808 07:01:46.432427 (+686967us) tablet_service.cc:3304] Waiting for operations to commit
0808 07:01:47.143566 (+711139us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1398101 microseconds
0808 07:01:47.143585 (+ 19us) tablet_service.cc:2945] Iterator created
0808 07:01:47.143625 (+ 40us) tablet_service.cc:2967] Iterator init: OK
0808 07:01:47.143628 (+ 3us) tablet_service.cc:3010] has_more: true
0808 07:01:47.143633 (+ 5us) tablet_service.cc:3027] Continuing scan request
0808 07:01:47.143645 (+ 12us) tablet_service.cc:3091] Found scanner 7af98fc7a9a945d2afdafaf20b538ae1 for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is 416794ab884c4d79868c423114541ee2
0808 07:01:47.143689 (+ 44us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":7808}
I20260808 07:01:47.143666 18659 rpcz_store.cc:276] 0808 07:01:45.595019 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:45.595065 (+ 46us) service_pool.cc:227] Handling call
0808 07:01:47.143572 (+1548507us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:45.703217 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:45.703233 (+ 16us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:45.703233 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:01:45.703234 (+ 1us) tablet.cc:660] Decoding operations
0808 07:01:45.703249 (+ 15us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:45.703252 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:45.703253 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:45.703264 (+ 11us) tablet.cc:699] Row locks acquired
0808 07:01:45.703265 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:01:45.703273 (+ 8us) write_op.cc:270] Start()
0808 07:01:46.432331 (+729058us) write_op.cc:276] Timestamp: P: 1786172505703228 usec, L: 0
0808 07:01:46.432333 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:01:46.432431 (+ 98us) log.cc:842] Serialized 354 byte log entry
0808 07:01:47.142992 (+710561us) op_driver.cc:444] REPLICATION: finished
0808 07:01:47.143119 (+ 127us) write_op.cc:301] APPLY: starting
0808 07:01:47.143124 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:47.143128 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:47.143132 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:47.143145 (+ 13us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:47.143149 (+ 4us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:47.143150 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:01:47.143168 (+ 18us) log.cc:842] Serialized 22 byte log entry
0808 07:01:47.143551 (+ 383us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:47.143553 (+ 2us) write_op.cc:454] Released schema lock
0808 07:01:47.143554 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":115,"mutex_wait_us":19,"num_ops":1,"prepare.queue_time_us":108035,"prepare.run_cpu_time_us":259,"prepare.run_wall_time_us":729276,"raft.queue_time_us":91,"raft.run_cpu_time_us":2,"raft.run_wall_time_us":1,"replication_time_us":710655,"thread_start_us":69,"threads_started":1}]]}
W20260808 07:01:47.143963 18478 scanner-internal.cc:458] Time spent opening tablet: real 1.399s user 0.000s sys 0.000s
I20260808 07:01:47.875083 18487 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:47.876041 18664 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=35, attempt_no=0}) took 1751 ms. Trace:
I20260808 07:01:47.876156 18664 rpcz_store.cc:276] 0808 07:01:46.124089 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:46.124135 (+ 46us) service_pool.cc:227] Handling call
0808 07:01:47.876030 (+1751895us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:46.432499 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:46.432514 (+ 15us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:46.432514 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:01:46.432516 (+ 2us) tablet.cc:660] Decoding operations
0808 07:01:46.432529 (+ 13us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:46.432532 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:46.432534 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:46.432544 (+ 10us) tablet.cc:699] Row locks acquired
0808 07:01:46.432545 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:01:46.432550 (+ 5us) write_op.cc:270] Start()
0808 07:01:46.809649 (+377099us) write_op.cc:276] Timestamp: P: 1786172506432502 usec, L: 0
0808 07:01:46.809652 (+ 3us) op_driver.cc:331] REPLICATION: starting
0808 07:01:46.809722 (+ 70us) log.cc:842] Serialized 354 byte log entry
0808 07:01:47.875709 (+1065987us) op_driver.cc:444] REPLICATION: finished
0808 07:01:47.875840 (+ 131us) write_op.cc:301] APPLY: starting
0808 07:01:47.875957 (+ 117us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:47.875961 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:47.875963 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:47.875975 (+ 12us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:47.875978 (+ 3us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:47.875979 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:01:47.875995 (+ 16us) log.cc:842] Serialized 22 byte log entry
0808 07:01:47.876011 (+ 16us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:47.876013 (+ 2us) write_op.cc:454] Released schema lock
0808 07:01:47.876014 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":119,"num_ops":1,"prepare.queue_time_us":308145,"prepare.run_cpu_time_us":301,"prepare.run_wall_time_us":377367,"replication_time_us":1066053,"thread_start_us":76,"threads_started":1}]]}
I20260808 07:01:47.980829 18539 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:48.001909 18539 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:48.002015 18539 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:48.002224 18539 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:48.002472 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:48.002466 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:48.002874 18041 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:48.003011 18539 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 07:01:48.003270 18478 scanner-internal.cc:458] Time spent opening tablet: real 0.709s user 0.000s sys 0.000s
W20260808 07:01:48.003304 18477 scanner-internal.cc:458] Time spent opening tablet: real 0.709s user 0.000s sys 0.000s
I20260808 07:01:48.586656 18487 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:48.586762 18487 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:48.587007 18487 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 910bd273924d4cb0adc0f7b65e454f19 (127.15.250.129:45605), 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273)
I20260808 07:01:48.587263 18111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "910bd273924d4cb0adc0f7b65e454f19" is_pre_election: true
I20260808 07:01:48.587378 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:48.587582 18322 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 221e532665ab4c7f9585be8ed849b6a9; no voters: 03f850c83dff43fdade5214b4dbee7a1, 910bd273924d4cb0adc0f7b65e454f19
I20260808 07:01:48.587723 18487 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:48.732092 18487 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:48.732251 18487 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:48.732496 18487 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 910bd273924d4cb0adc0f7b65e454f19 (127.15.250.129:45605), 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273)
I20260808 07:01:48.732757 18112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "910bd273924d4cb0adc0f7b65e454f19" is_pre_election: true
I20260808 07:01:48.732766 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:48.733065 18322 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 221e532665ab4c7f9585be8ed849b6a9; no voters: 03f850c83dff43fdade5214b4dbee7a1, 910bd273924d4cb0adc0f7b65e454f19
W20260808 07:01:48.733572 18478 scanner-internal.cc:458] Time spent opening tablet: real 0.580s user 0.000s sys 0.000s
W20260808 07:01:48.733572 18477 scanner-internal.cc:458] Time spent opening tablet: real 0.580s user 0.000s sys 0.000s
I20260808 07:01:48.909420 18487 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:49.235225 18674 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:49.235335 18674 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:49.235543 18674 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:49.235656 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:49.235700 18388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:49.235936 18041 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:49.236065 18674 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:49.390653 18674 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:49.390760 18674 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:49.390954 18674 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:49.391142 18388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:49.391139 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:49.391419 18042 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:49.391518 18674 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:49.629920 18679 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:49.630039 18679 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:49.630373 18679 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 910bd273924d4cb0adc0f7b65e454f19 (127.15.250.129:45605), 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273)
I20260808 07:01:49.630546 18112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "910bd273924d4cb0adc0f7b65e454f19" is_pre_election: true
I20260808 07:01:49.630745 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:49.631078 18322 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 221e532665ab4c7f9585be8ed849b6a9; no voters: 03f850c83dff43fdade5214b4dbee7a1, 910bd273924d4cb0adc0f7b65e454f19
I20260808 07:01:49.631250 18679 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 07:01:49.631312 18477 scanner-internal.cc:458] Time spent opening tablet: real 0.747s user 0.000s sys 0.000s
W20260808 07:01:49.631350 18478 scanner-internal.cc:458] Time spent opening tablet: real 0.747s user 0.000s sys 0.000s
I20260808 07:01:50.373857 18674 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:50.373967 18674 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:50.374167 18674 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:50.374446 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:50.376546 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:50.376858 18041 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:50.817209 18674 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:50.817998 18687 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=40, attempt_no=0}) took 1907 ms. Trace:
I20260808 07:01:50.818177 18207 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60832 (request call id 90) took 1036 ms. Trace:
I20260808 07:01:50.818182 18208 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60832 (request call id 91) took 1036 ms. Trace:
I20260808 07:01:50.818140 18687 rpcz_store.cc:276] 0808 07:01:48.910532 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:48.910552 (+ 20us) service_pool.cc:227] Handling call
0808 07:01:50.817987 (+1907435us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:49.501250 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:49.501266 (+ 16us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:49.501266 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:01:49.501267 (+ 1us) tablet.cc:660] Decoding operations
0808 07:01:49.501281 (+ 14us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:49.501284 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:49.501286 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:49.501297 (+ 11us) tablet.cc:699] Row locks acquired
0808 07:01:49.501297 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:01:49.501303 (+ 6us) write_op.cc:270] Start()
0808 07:01:50.359402 (+858099us) write_op.cc:276] Timestamp: P: 1786172509501261 usec, L: 0
0808 07:01:50.359403 (+ 1us) op_driver.cc:331] REPLICATION: starting
0808 07:01:50.359503 (+ 100us) log.cc:842] Serialized 354 byte log entry
0808 07:01:50.817757 (+458254us) op_driver.cc:444] REPLICATION: finished
0808 07:01:50.817851 (+ 94us) write_op.cc:301] APPLY: starting
0808 07:01:50.817855 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:50.817860 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:50.817863 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:50.817879 (+ 16us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:50.817890 (+ 11us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:50.817891 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:01:50.817924 (+ 33us) log.cc:842] Serialized 22 byte log entry
0808 07:01:50.817962 (+ 38us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:50.817965 (+ 3us) write_op.cc:454] Released schema lock
0808 07:01:50.817965 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":80,"num_ops":1,"prepare.queue_time_us":590612,"prepare.run_cpu_time_us":216,"prepare.run_wall_time_us":858284,"raft.queue_time_us":15,"raft.run_cpu_time_us":35,"raft.run_wall_time_us":36,"replication_time_us":458350,"thread_start_us":55,"threads_started":1}]]}
I20260808 07:01:50.818491 18208 rpcz_store.cc:276] 0808 07:01:49.781929 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:49.781971 (+ 42us) service_pool.cc:227] Handling call
0808 07:01:49.782013 (+ 42us) tablet_service.cc:2780] Created scanner a5d08149619341ad8673bfcd4681630c for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is f02f04ded421430885fa1f67ec61cbf4
0808 07:01:49.782052 (+ 39us) tablet_service.cc:2920] Creating iterator
0808 07:01:49.782057 (+ 5us) tablet_service.cc:3297] Waiting safe time to advance
0808 07:01:50.359471 (+577414us) tablet_service.cc:3304] Waiting for operations to commit
0808 07:01:50.818044 (+458573us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1035984 microseconds
0808 07:01:50.818059 (+ 15us) tablet_service.cc:2945] Iterator created
0808 07:01:50.818103 (+ 44us) tablet_service.cc:2967] Iterator init: OK
0808 07:01:50.818107 (+ 4us) tablet_service.cc:3010] has_more: true
0808 07:01:50.818112 (+ 5us) tablet_service.cc:3027] Continuing scan request
0808 07:01:50.818124 (+ 12us) tablet_service.cc:3091] Found scanner a5d08149619341ad8673bfcd4681630c for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is f02f04ded421430885fa1f67ec61cbf4
0808 07:01:50.818179 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
I20260808 07:01:50.818329 18207 rpcz_store.cc:276] 0808 07:01:49.781880 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:49.781925 (+ 45us) service_pool.cc:227] Handling call
0808 07:01:49.781968 (+ 43us) tablet_service.cc:2780] Created scanner d2de197eb8b440cdbf1cf307115ced0f for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is 34efeb06b4ce497fa8a0659ecf40f2fb
0808 07:01:49.782023 (+ 55us) tablet_service.cc:2920] Creating iterator
0808 07:01:49.782032 (+ 9us) tablet_service.cc:3297] Waiting safe time to advance
0808 07:01:50.359479 (+577447us) tablet_service.cc:3304] Waiting for operations to commit
0808 07:01:50.817975 (+458496us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1035938 microseconds
0808 07:01:50.817995 (+ 20us) tablet_service.cc:2945] Iterator created
0808 07:01:50.818054 (+ 59us) tablet_service.cc:2967] Iterator init: OK
0808 07:01:50.818059 (+ 5us) tablet_service.cc:3010] has_more: true
0808 07:01:50.818077 (+ 18us) tablet_service.cc:3027] Continuing scan request
0808 07:01:50.818093 (+ 16us) tablet_service.cc:3091] Found scanner d2de197eb8b440cdbf1cf307115ced0f for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is 34efeb06b4ce497fa8a0659ecf40f2fb
0808 07:01:50.818169 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
W20260808 07:01:50.818872 18478 scanner-internal.cc:458] Time spent opening tablet: real 1.037s user 0.000s sys 0.000s
W20260808 07:01:50.818872 18477 scanner-internal.cc:458] Time spent opening tablet: real 1.037s user 0.000s sys 0.000s
I20260808 07:01:50.863379 18683 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:50.863509 18683 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:50.863719 18683 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 910bd273924d4cb0adc0f7b65e454f19 (127.15.250.129:45605), 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273)
I20260808 07:01:50.863945 18112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "910bd273924d4cb0adc0f7b65e454f19" is_pre_election: true
I20260808 07:01:50.864039 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:50.864352 18322 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 221e532665ab4c7f9585be8ed849b6a9; no voters: 03f850c83dff43fdade5214b4dbee7a1, 910bd273924d4cb0adc0f7b65e454f19
I20260808 07:01:51.338548 18683 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:51.339428 18693 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=41, attempt_no=0}) took 1707 ms. Trace:
I20260808 07:01:51.339545 18693 rpcz_store.cc:276] 0808 07:01:49.631586 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:49.631639 (+ 53us) service_pool.cc:227] Handling call
0808 07:01:51.339416 (+1707777us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:50.359543 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:50.359560 (+ 17us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:50.359561 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:01:50.359561 (+ 0us) tablet.cc:660] Decoding operations
0808 07:01:50.359575 (+ 14us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:50.359577 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:50.359579 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:50.359590 (+ 11us) tablet.cc:699] Row locks acquired
0808 07:01:50.359590 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:01:50.359596 (+ 6us) write_op.cc:270] Start()
0808 07:01:50.838694 (+479098us) write_op.cc:276] Timestamp: P: 1786172510359568 usec, L: 0
0808 07:01:50.838695 (+ 1us) op_driver.cc:331] REPLICATION: starting
0808 07:01:50.838776 (+ 81us) log.cc:842] Serialized 354 byte log entry
0808 07:01:51.339192 (+500416us) op_driver.cc:444] REPLICATION: finished
0808 07:01:51.339315 (+ 123us) write_op.cc:301] APPLY: starting
0808 07:01:51.339320 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:51.339326 (+ 6us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:51.339330 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:51.339343 (+ 13us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:51.339348 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:51.339349 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:01:51.339369 (+ 20us) log.cc:842] Serialized 22 byte log entry
0808 07:01:51.339391 (+ 22us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:51.339393 (+ 2us) write_op.cc:454] Released schema lock
0808 07:01:51.339394 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":110,"num_ops":1,"prepare.queue_time_us":727822,"prepare.run_cpu_time_us":207,"prepare.run_wall_time_us":479276,"raft.queue_time_us":36,"raft.run_cpu_time_us":23,"raft.run_wall_time_us":24,"replication_time_us":500492,"spinlock_wait_cycles":39680,"thread_start_us":43,"threads_started":1}]]}
I20260808 07:01:51.838593 18674 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:51.838718 18674 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:51.838943 18674 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:51.839133 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:51.839145 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:51.839524 18042 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:51.839707 18674 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:51.862854 18683 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:51.862974 18683 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:51.863196 18683 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 910bd273924d4cb0adc0f7b65e454f19 (127.15.250.129:45605), 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273)
I20260808 07:01:51.863422 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 44 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:51.863441 18112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 44 } } ignore_live_leader: false dest_uuid: "910bd273924d4cb0adc0f7b65e454f19" is_pre_election: true
I20260808 07:01:51.863750 18322 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 221e532665ab4c7f9585be8ed849b6a9; no voters: 03f850c83dff43fdade5214b4dbee7a1, 910bd273924d4cb0adc0f7b65e454f19
I20260808 07:01:51.863870 18683 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:51.863940 18699 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=42, attempt_no=0}) took 1044 ms. Trace:
I20260808 07:01:51.863986 18699 rpcz_store.cc:276] 0808 07:01:50.818993 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:50.819034 (+ 41us) service_pool.cc:227] Handling call
0808 07:01:51.863933 (+1044899us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:50.838828 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:50.838844 (+ 16us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:50.838844 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:01:50.838844 (+ 0us) tablet.cc:660] Decoding operations
0808 07:01:50.838859 (+ 15us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:50.838861 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:50.838863 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:50.838873 (+ 10us) tablet.cc:699] Row locks acquired
0808 07:01:50.838874 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:01:50.838880 (+ 6us) write_op.cc:270] Start()
0808 07:01:50.919023 (+ 80143us) write_op.cc:276] Timestamp: P: 1786172510838839 usec, L: 0
0808 07:01:50.919026 (+ 3us) op_driver.cc:331] REPLICATION: starting
0808 07:01:50.919119 (+ 93us) log.cc:842] Serialized 354 byte log entry
0808 07:01:51.863490 (+944371us) op_driver.cc:444] REPLICATION: finished
0808 07:01:51.863582 (+ 92us) write_op.cc:301] APPLY: starting
0808 07:01:51.863586 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:51.863591 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:51.863594 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:51.863606 (+ 12us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:51.863611 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:51.863612 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:01:51.863631 (+ 19us) log.cc:842] Serialized 22 byte log entry
0808 07:01:51.863908 (+ 277us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:51.863912 (+ 4us) write_op.cc:454] Released schema lock
0808 07:01:51.863912 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":81,"num_ops":1,"prepare.queue_time_us":19659,"prepare.run_cpu_time_us":251,"prepare.run_wall_time_us":80330,"replication_time_us":944459,"thread_start_us":34,"threads_started":1}]]}
W20260808 07:01:51.864126 18478 scanner-internal.cc:458] Time spent opening tablet: real 0.895s user 0.000s sys 0.000s
W20260808 07:01:51.864457 18477 scanner-internal.cc:458] Time spent opening tablet: real 0.895s user 0.000s sys 0.000s
I20260808 07:01:52.380335 18674 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:52.380445 18674 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:52.380702 18674 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:52.380991 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 44 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:52.380991 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 44 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:52.381278 18042 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:52.661746 18674 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:52.662596 18705 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=43, attempt_no=0}) took 1322 ms. Trace:
I20260808 07:01:52.662714 18705 rpcz_store.cc:276] 0808 07:01:51.340242 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:51.340327 (+ 85us) service_pool.cc:227] Handling call
0808 07:01:52.662587 (+1322260us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:51.340547 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:51.340558 (+ 11us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:51.340559 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:01:51.340559 (+ 0us) tablet.cc:660] Decoding operations
0808 07:01:51.340568 (+ 9us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:51.340570 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:51.340571 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:51.340577 (+ 6us) tablet.cc:699] Row locks acquired
0808 07:01:51.340578 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:01:51.340584 (+ 6us) write_op.cc:270] Start()
0808 07:01:51.923700 (+583116us) write_op.cc:276] Timestamp: P: 1786172511340544 usec, L: 0
0808 07:01:51.923702 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:01:51.923757 (+ 55us) log.cc:842] Serialized 354 byte log entry
0808 07:01:52.662354 (+738597us) op_driver.cc:444] REPLICATION: finished
0808 07:01:52.662502 (+ 148us) write_op.cc:301] APPLY: starting
0808 07:01:52.662506 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:52.662510 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:52.662513 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:52.662527 (+ 14us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:52.662532 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:52.662532 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:01:52.662549 (+ 17us) log.cc:842] Serialized 22 byte log entry
0808 07:01:52.662567 (+ 18us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:52.662569 (+ 2us) write_op.cc:454] Released schema lock
0808 07:01:52.662570 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":134,"num_ops":1,"prepare.queue_time_us":60,"prepare.run_cpu_time_us":195,"prepare.run_wall_time_us":583272,"raft.queue_time_us":45,"raft.run_cpu_time_us":39,"raft.run_wall_time_us":41,"replication_time_us":738648,"thread_start_us":105,"threads_started":1}]]}
I20260808 07:01:52.693943 18674 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:52.694051 18674 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:52.694370 18674 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:52.694491 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:52.694550 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:52.694885 18042 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:52.695005 18674 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 07:01:52.695251 18478 scanner-internal.cc:458] Time spent opening tablet: real 0.681s user 0.000s sys 0.000s
W20260808 07:01:52.695251 18477 scanner-internal.cc:458] Time spent opening tablet: real 0.681s user 0.000s sys 0.000s
I20260808 07:01:52.861510 18683 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:52.861606 18683 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:52.861809 18683 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 910bd273924d4cb0adc0f7b65e454f19 (127.15.250.129:45605), 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273)
I20260808 07:01:52.861980 18111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "910bd273924d4cb0adc0f7b65e454f19" is_pre_election: true
I20260808 07:01:52.862021 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:52.862273 18322 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 221e532665ab4c7f9585be8ed849b6a9; no voters: 03f850c83dff43fdade5214b4dbee7a1, 910bd273924d4cb0adc0f7b65e454f19
I20260808 07:01:53.857880 18683 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:54.166914 18683 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:54.167033 18683 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:54.167268 18683 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 910bd273924d4cb0adc0f7b65e454f19 (127.15.250.129:45605), 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273)
I20260808 07:01:54.167488 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:54.167490 18112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "910bd273924d4cb0adc0f7b65e454f19" is_pre_election: true
I20260808 07:01:54.167785 18322 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 221e532665ab4c7f9585be8ed849b6a9; no voters: 03f850c83dff43fdade5214b4dbee7a1, 910bd273924d4cb0adc0f7b65e454f19
I20260808 07:01:54.167806 18714 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=45, attempt_no=0}) took 1504 ms. Trace:
I20260808 07:01:54.167853 18714 rpcz_store.cc:276] 0808 07:01:52.663240 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:52.663265 (+ 25us) service_pool.cc:227] Handling call
0808 07:01:54.167797 (+1504532us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:52.663381 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:52.663391 (+ 10us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:52.663392 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:01:52.663392 (+ 0us) tablet.cc:660] Decoding operations
0808 07:01:52.663401 (+ 9us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:52.663407 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:52.663408 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:52.663415 (+ 7us) tablet.cc:699] Row locks acquired
0808 07:01:52.663415 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:01:52.663422 (+ 7us) write_op.cc:270] Start()
0808 07:01:53.354530 (+691108us) write_op.cc:276] Timestamp: P: 1786172512663369 usec, L: 0
0808 07:01:53.354531 (+ 1us) op_driver.cc:331] REPLICATION: starting
0808 07:01:53.354629 (+ 98us) log.cc:842] Serialized 354 byte log entry
0808 07:01:54.167590 (+812961us) op_driver.cc:444] REPLICATION: finished
0808 07:01:54.167688 (+ 98us) write_op.cc:301] APPLY: starting
0808 07:01:54.167692 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:54.167696 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:54.167699 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:54.167713 (+ 14us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:54.167718 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:54.167718 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:01:54.167736 (+ 18us) log.cc:842] Serialized 22 byte log entry
0808 07:01:54.167774 (+ 38us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:54.167777 (+ 3us) write_op.cc:454] Released schema lock
0808 07:01:54.167778 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":86,"num_ops":1,"prepare.queue_time_us":30,"prepare.run_cpu_time_us":208,"prepare.run_wall_time_us":691284,"raft.queue_time_us":68,"raft.run_cpu_time_us":40,"raft.run_wall_time_us":41,"replication_time_us":813055,"thread_start_us":44,"threads_started":1}]]}
I20260808 07:01:54.167969 18683 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:54.168020 18208 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60832 (request call id 101) took 1322 ms. Trace:
I20260808 07:01:54.168066 18208 rpcz_store.cc:276] 0808 07:01:52.845707 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:52.845753 (+ 46us) service_pool.cc:227] Handling call
0808 07:01:52.845800 (+ 47us) tablet_service.cc:2780] Created scanner 40acb3d9a7fa4291b90097a61c6acc92 for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is b5dd75311c39493f99f9724cbfc8427c
0808 07:01:52.845838 (+ 38us) tablet_service.cc:2920] Creating iterator
0808 07:01:52.845844 (+ 6us) tablet_service.cc:3297] Waiting safe time to advance
0808 07:01:53.354605 (+508761us) tablet_service.cc:3304] Waiting for operations to commit
0808 07:01:54.167860 (+813255us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1322012 microseconds
0808 07:01:54.167875 (+ 15us) tablet_service.cc:2945] Iterator created
0808 07:01:54.167919 (+ 44us) tablet_service.cc:2967] Iterator init: OK
0808 07:01:54.167923 (+ 4us) tablet_service.cc:3010] has_more: true
0808 07:01:54.167927 (+ 4us) tablet_service.cc:3027] Continuing scan request
0808 07:01:54.167951 (+ 24us) tablet_service.cc:3091] Found scanner 40acb3d9a7fa4291b90097a61c6acc92 for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is b5dd75311c39493f99f9724cbfc8427c
0808 07:01:54.168013 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":11648}
W20260808 07:01:54.168365 18477 scanner-internal.cc:458] Time spent opening tablet: real 1.323s user 0.000s sys 0.000s
I20260808 07:01:54.167940 18207 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60832 (request call id 102) took 1322 ms. Trace:
I20260808 07:01:54.168497 18207 rpcz_store.cc:276] 0808 07:01:52.845727 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:52.845767 (+ 40us) service_pool.cc:227] Handling call
0808 07:01:52.845809 (+ 42us) tablet_service.cc:2780] Created scanner 32b065d6336348b8bf504be531d29e9f for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is c7abaa18ea534e8db0bb7392143a6780
0808 07:01:52.845843 (+ 34us) tablet_service.cc:2920] Creating iterator
0808 07:01:52.845851 (+ 8us) tablet_service.cc:3297] Waiting safe time to advance
0808 07:01:53.354581 (+508730us) tablet_service.cc:3304] Waiting for operations to commit
0808 07:01:54.167785 (+813204us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1321930 microseconds
0808 07:01:54.167803 (+ 18us) tablet_service.cc:2945] Iterator created
0808 07:01:54.167854 (+ 51us) tablet_service.cc:2967] Iterator init: OK
0808 07:01:54.167859 (+ 5us) tablet_service.cc:3010] has_more: true
0808 07:01:54.167865 (+ 6us) tablet_service.cc:3027] Continuing scan request
0808 07:01:54.167878 (+ 13us) tablet_service.cc:3091] Found scanner 32b065d6336348b8bf504be531d29e9f for tablet fcb881fce4ea471890e23b30bc4f2eb4, query id is c7abaa18ea534e8db0bb7392143a6780
0808 07:01:54.167938 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":8064}
W20260808 07:01:54.168709 18478 scanner-internal.cc:458] Time spent opening tablet: real 1.323s user 0.000s sys 0.000s
I20260808 07:01:54.323477 18709 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:54.323593 18709 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:54.323829 18709 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:54.323937 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:54.324059 18388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:54.324326 18041 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:54.324426 18709 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:54.757542 18709 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:54.757650 18709 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:54.757846 18709 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:54.758106 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:54.758177 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:54.758399 18041 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:54.758556 18709 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:54.758631 18718 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=46, attempt_no=0}) took 2063 ms. Trace:
I20260808 07:01:54.758682 18718 rpcz_store.cc:276] 0808 07:01:52.695348 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:52.695387 (+ 39us) service_pool.cc:227] Handling call
0808 07:01:54.758624 (+2063237us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:53.354677 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:53.354692 (+ 15us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:53.354693 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:01:53.354694 (+ 1us) tablet.cc:660] Decoding operations
0808 07:01:53.354707 (+ 13us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:53.354709 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:53.354710 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:53.354721 (+ 11us) tablet.cc:699] Row locks acquired
0808 07:01:53.354721 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:01:53.354727 (+ 6us) write_op.cc:270] Start()
0808 07:01:54.330836 (+976109us) write_op.cc:276] Timestamp: P: 1786172513354697 usec, L: 0
0808 07:01:54.330837 (+ 1us) op_driver.cc:331] REPLICATION: starting
0808 07:01:54.330915 (+ 78us) log.cc:842] Serialized 354 byte log entry
0808 07:01:54.758158 (+427243us) op_driver.cc:444] REPLICATION: finished
0808 07:01:54.758525 (+ 367us) write_op.cc:301] APPLY: starting
0808 07:01:54.758528 (+ 3us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:54.758531 (+ 3us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:54.758534 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:54.758544 (+ 10us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:54.758548 (+ 4us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:54.758549 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:01:54.758564 (+ 15us) log.cc:842] Serialized 22 byte log entry
0808 07:01:54.758600 (+ 36us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:54.758603 (+ 3us) write_op.cc:454] Released schema lock
0808 07:01:54.758604 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":184,"num_ops":1,"prepare.queue_time_us":659170,"prepare.run_cpu_time_us":209,"prepare.run_wall_time_us":976289,"raft.queue_time_us":51,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":167,"replication_time_us":427317,"spinlock_wait_cycles":9984,"thread_start_us":127,"threads_started":1}]]}
I20260808 07:01:54.784631 18683 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:54.784786 18683 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:54.784981 18683 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 910bd273924d4cb0adc0f7b65e454f19 (127.15.250.129:45605), 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273)
I20260808 07:01:54.785154 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:54.785218 18111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "910bd273924d4cb0adc0f7b65e454f19" is_pre_election: true
I20260808 07:01:54.785432 18322 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 221e532665ab4c7f9585be8ed849b6a9; no voters: 03f850c83dff43fdade5214b4dbee7a1, 910bd273924d4cb0adc0f7b65e454f19
I20260808 07:01:54.785503 18683 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:54.988914 18683 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:54.989005 18683 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:54.989187 18683 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 910bd273924d4cb0adc0f7b65e454f19 (127.15.250.129:45605), 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273)
I20260808 07:01:54.989415 18112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 49 } } ignore_live_leader: false dest_uuid: "910bd273924d4cb0adc0f7b65e454f19" is_pre_election: true
I20260808 07:01:54.989501 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 49 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:54.989724 18322 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 221e532665ab4c7f9585be8ed849b6a9; no voters: 03f850c83dff43fdade5214b4dbee7a1, 910bd273924d4cb0adc0f7b65e454f19
I20260808 07:01:55.777124 18683 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:55.777956 18725 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=48, attempt_no=0}) took 1018 ms. Trace:
I20260808 07:01:55.778090 18725 rpcz_store.cc:276] 0808 07:01:54.759261 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:54.759298 (+ 37us) service_pool.cc:227] Handling call
0808 07:01:55.777944 (+1018646us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:54.885233 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:54.885250 (+ 17us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:54.885250 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:01:54.885251 (+ 1us) tablet.cc:660] Decoding operations
0808 07:01:54.885265 (+ 14us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:54.885269 (+ 4us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:54.885270 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:54.885282 (+ 12us) tablet.cc:699] Row locks acquired
0808 07:01:54.885282 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:01:54.885291 (+ 9us) write_op.cc:270] Start()
0808 07:01:54.982389 (+ 97098us) write_op.cc:276] Timestamp: P: 1786172514885261 usec, L: 0
0808 07:01:54.982392 (+ 3us) op_driver.cc:331] REPLICATION: starting
0808 07:01:54.982535 (+ 143us) log.cc:842] Serialized 354 byte log entry
0808 07:01:55.777685 (+795150us) op_driver.cc:444] REPLICATION: finished
0808 07:01:55.777821 (+ 136us) write_op.cc:301] APPLY: starting
0808 07:01:55.777826 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:55.777830 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:55.777832 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:55.777854 (+ 22us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:55.777859 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:55.777860 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:01:55.777881 (+ 21us) log.cc:842] Serialized 22 byte log entry
0808 07:01:55.777919 (+ 38us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:55.777922 (+ 3us) write_op.cc:454] Released schema lock
0808 07:01:55.777923 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":120,"num_ops":1,"prepare.queue_time_us":125815,"prepare.run_cpu_time_us":257,"prepare.run_wall_time_us":97323,"replication_time_us":795288,"thread_start_us":57,"threads_started":1}]]}
W20260808 07:01:55.778398 18478 scanner-internal.cc:458] Time spent opening tablet: real 0.869s user 0.000s sys 0.000s
W20260808 07:01:55.778398 18477 scanner-internal.cc:458] Time spent opening tablet: real 0.869s user 0.000s sys 0.000s
I20260808 07:01:55.784870 18709 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:55.784966 18709 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:55.785256 18709 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:55.785410 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 49 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:55.785422 18388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 49 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:55.785691 18041 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:56.218091 18683 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:56.218187 18683 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:56.218411 18683 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 910bd273924d4cb0adc0f7b65e454f19 (127.15.250.129:45605), 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273)
I20260808 07:01:56.218580 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 51 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:56.218595 18112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "221e532665ab4c7f9585be8ed849b6a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 51 } } ignore_live_leader: false dest_uuid: "910bd273924d4cb0adc0f7b65e454f19" is_pre_election: true
I20260808 07:01:56.218807 18725 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60832 (ReqId={client: 3f3fd70be2f94fae921bf0c4df52d2cd, seq_no=49, attempt_no=0}) took 1228 ms. Trace:
I20260808 07:01:56.218818 18322 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 221e532665ab4c7f9585be8ed849b6a9; no voters: 03f850c83dff43fdade5214b4dbee7a1, 910bd273924d4cb0adc0f7b65e454f19
I20260808 07:01:56.218863 18725 rpcz_store.cc:276] 0808 07:01:54.990079 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:54.990119 (+ 40us) service_pool.cc:227] Handling call
0808 07:01:56.218800 (+1228681us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:54.990262 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fcb881fce4ea471890e23b30bc4f2eb4
0808 07:01:54.990271 (+ 9us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:54.990271 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:01:54.990271 (+ 0us) tablet.cc:660] Decoding operations
0808 07:01:54.990280 (+ 9us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:54.990281 (+ 1us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:54.990282 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:54.990288 (+ 6us) tablet.cc:699] Row locks acquired
0808 07:01:54.990289 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:01:54.990294 (+ 5us) write_op.cc:270] Start()
0808 07:01:55.822411 (+832117us) write_op.cc:276] Timestamp: P: 1786172514990257 usec, L: 0
0808 07:01:55.822413 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:01:55.822494 (+ 81us) log.cc:842] Serialized 354 byte log entry
0808 07:01:56.218554 (+396060us) op_driver.cc:444] REPLICATION: finished
0808 07:01:56.218703 (+ 149us) write_op.cc:301] APPLY: starting
0808 07:01:56.218707 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:56.218713 (+ 6us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:56.218716 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:56.218729 (+ 13us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:56.218735 (+ 6us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:56.218736 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:01:56.218755 (+ 19us) log.cc:842] Serialized 22 byte log entry
0808 07:01:56.218777 (+ 22us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:56.218780 (+ 3us) write_op.cc:454] Released schema lock
0808 07:01:56.218781 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":111,"num_ops":1,"prepare.queue_time_us":30,"prepare.run_cpu_time_us":185,"prepare.run_wall_time_us":832267,"raft.queue_time_us":15,"raft.run_cpu_time_us":15,"raft.run_wall_time_us":15,"replication_time_us":396137,"spinlock_wait_cycles":23936}]]}
I20260808 07:01:56.219026 18683 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:56.250936 18367 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260808 07:01:56.251892 18225 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260808 07:01:56.422981 18709 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:56.424288 18091 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260808 07:01:57.198943 18709 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:57.199138 18709 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:57.199376 18709 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273), 221e532665ab4c7f9585be8ed849b6a9 (127.15.250.131:44581)
I20260808 07:01:57.199502 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 51 } } ignore_live_leader: false dest_uuid: "03f850c83dff43fdade5214b4dbee7a1" is_pre_election: true
I20260808 07:01:57.199563 18388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb881fce4ea471890e23b30bc4f2eb4" candidate_uuid: "910bd273924d4cb0adc0f7b65e454f19" candidate_term: 2 candidate_status { last_received { term: 1 index: 51 } } ignore_live_leader: false dest_uuid: "221e532665ab4c7f9585be8ed849b6a9" is_pre_election: true
I20260808 07:01:57.199855 18041 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910bd273924d4cb0adc0f7b65e454f19; no voters: 03f850c83dff43fdade5214b4dbee7a1, 221e532665ab4c7f9585be8ed849b6a9
I20260808 07:01:57.199970 18709 raft_consensus.cc:2726] T fcb881fce4ea471890e23b30bc4f2eb4 P 910bd273924d4cb0adc0f7b65e454f19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
298377829a064c73bfa7cba5ab5bd72c | 127.15.250.190:44265 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.250.148:42451 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
03f850c83dff43fdade5214b4dbee7a1 | 127.15.250.130:43273 | HEALTHY | <none> | 1 | 0
221e532665ab4c7f9585be8ed849b6a9 | 127.15.250.131:44581 | HEALTHY | <none> | 0 | 0
910bd273924d4cb0adc0f7b65e454f19 | 127.15.250.129:45605 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
inject_latency_ms_before_starting_op | 1000 | hidden,advanced | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.15.250.129 | experimental | 127.15.250.129:45605
local_ip_for_outbound_sockets | 127.15.250.130 | experimental | 127.15.250.130:43273
local_ip_for_outbound_sockets | 127.15.250.131 | experimental | 127.15.250.131:44581
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.15.250.129:45605
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.15.250.130:43273
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.15.250.131:44581
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.250.148:42451 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260808 07:01:57.207511 16362 log_verifier.cc:126] Checking tablet fcb881fce4ea471890e23b30bc4f2eb4
I20260808 07:01:57.228804 16362 log_verifier.cc:177] Verified matching terms for 51 ops in tablet fcb881fce4ea471890e23b30bc4f2eb4
I20260808 07:01:57.239449 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 18024
W20260808 07:01:57.246450 18182 connection.cc:535] client connection to 127.15.250.129:45605 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 07:01:57.246537 18182 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 1 similar messages]
I20260808 07:01:57.246929 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 18164
I20260808 07:01:57.253006 18776 raft_consensus.cc:482] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03f850c83dff43fdade5214b4dbee7a1)
I20260808 07:01:57.253091 18776 raft_consensus.cc:504] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910bd273924d4cb0adc0f7b65e454f19" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45605 } } peers { permanent_uuid: "03f850c83dff43fdade5214b4dbee7a1" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43273 } } peers { permanent_uuid: "221e532665ab4c7f9585be8ed849b6a9" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44581 } }
I20260808 07:01:57.253255 18776 leader_election.cc:288] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 910bd273924d4cb0adc0f7b65e454f19 (127.15.250.129:45605), 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273)
W20260808 07:01:57.253582 18322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.250.129:45605: connect: Connection refused (error 111)
W20260808 07:01:57.253975 18322 leader_election.cc:334] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 910bd273924d4cb0adc0f7b65e454f19 (127.15.250.129:45605): Network error: Client connection negotiation failed: client connection to 127.15.250.129:45605: connect: Connection refused (error 111)
W20260808 07:01:57.255273 18322 connection.cc:535] client connection to 127.15.250.130:43273 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 07:01:57.255697 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 18304
W20260808 07:01:57.255736 18322 leader_election.cc:334] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 03f850c83dff43fdade5214b4dbee7a1 (127.15.250.130:43273): Network error: Client connection negotiation failed: client connection to 127.15.250.130:43273: connect: Connection refused (error 111)
I20260808 07:01:57.255779 18322 leader_election.cc:302] T fcb881fce4ea471890e23b30bc4f2eb4 P 221e532665ab4c7f9585be8ed849b6a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 221e532665ab4c7f9585be8ed849b6a9; no voters: 03f850c83dff43fdade5214b4dbee7a1, 910bd273924d4cb0adc0f7b65e454f19
I20260808 07:01:57.261390 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 17927
2026-08-08T07:01:57Z chronyd exiting
[ OK ] RaftConsensusElectionITest.ChurnyElections (32592 ms)
[ RUN ] RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency
I20260808 07:01:57.285172 16362 test_util.cc:268] Using random seed: -1301906130
I20260808 07:01:57.285777 16362 ts_itest-base.cc:112] Starting cluster with:
I20260808 07:01:57.285846 16362 ts_itest-base.cc:113] --------------
I20260808 07:01:57.285864 16362 ts_itest-base.cc:114] 3 tablet servers
I20260808 07:01:57.285883 16362 ts_itest-base.cc:115] 3 replicas per TS
I20260808 07:01:57.285907 16362 ts_itest-base.cc:116] --------------
2026-08-08T07:01:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:01:57Z Disabled control of system clock
I20260808 07:01:57.294927 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.250.190:35249
--webserver_interface=127.15.250.190
--webserver_port=0
--builtin_ntp_servers=127.15.250.148:33287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.250.190:35249 with env {}
W20260808 07:01:57.376324 18787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:57.376578 18787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:57.376612 18787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:57.377776 18787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:01:57.377847 18787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:57.377871 18787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:01:57.377904 18787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:01:57.379259 18787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:33287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.250.190:35249
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.250.190:35249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.250.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.18787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:01:57.379534 18787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:57.379829 18787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:57.384841 18792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:57.385015 18787 server_base.cc:1034] running on GCE node
W20260808 07:01:57.384861 18793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:57.384900 18795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:57.385416 18787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:57.385766 18787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:57.386926 18787 hybrid_clock.cc:648] HybridClock initialized: now 1786172517386923 us; error 46 us; skew 500 ppm
I20260808 07:01:57.388201 18787 webserver.cc:460] Webserver started at http://127.15.250.190:46379/ using document root <none> and password file <none>
I20260808 07:01:57.388477 18787 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:57.388548 18787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:57.388691 18787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:57.389698 18787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "99990c52528042d783bfc0b3815c0715"
format_stamp: "Formatted at 2026-08-08 07:01:57 on dist-test-slave-0b58"
I20260808 07:01:57.390071 18787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "99990c52528042d783bfc0b3815c0715"
format_stamp: "Formatted at 2026-08-08 07:01:57 on dist-test-slave-0b58"
I20260808 07:01:57.391429 18787 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 07:01:57.392225 18801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:57.392426 18787 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:01:57.392504 18787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
uuid: "99990c52528042d783bfc0b3815c0715"
format_stamp: "Formatted at 2026-08-08 07:01:57 on dist-test-slave-0b58"
I20260808 07:01:57.392556 18787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:57.402242 18787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:57.402496 18787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:57.402599 18787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:57.406649 18787 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.190:35249
I20260808 07:01:57.406683 18858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.190:35249 every 8 connection(s)
I20260808 07:01:57.407038 18787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260808 07:01:57.407675 18859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:57.409304 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 18787
I20260808 07:01:57.409381 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260808 07:01:57.411022 18859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99990c52528042d783bfc0b3815c0715: Bootstrap starting.
I20260808 07:01:57.411774 18859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99990c52528042d783bfc0b3815c0715: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:57.412106 18859 log.cc:824] T 00000000000000000000000000000000 P 99990c52528042d783bfc0b3815c0715: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:57.412936 18859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99990c52528042d783bfc0b3815c0715: No bootstrap required, opened a new log
I20260808 07:01:57.414599 18859 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99990c52528042d783bfc0b3815c0715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99990c52528042d783bfc0b3815c0715" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 35249 } }
I20260808 07:01:57.414785 18859 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99990c52528042d783bfc0b3815c0715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:01:57.414836 18859 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99990c52528042d783bfc0b3815c0715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99990c52528042d783bfc0b3815c0715, State: Initialized, Role: FOLLOWER
I20260808 07:01:57.414973 18859 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99990c52528042d783bfc0b3815c0715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99990c52528042d783bfc0b3815c0715" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 35249 } }
I20260808 07:01:57.415066 18859 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99990c52528042d783bfc0b3815c0715 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:01:57.415102 18859 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99990c52528042d783bfc0b3815c0715 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:01:57.415154 18859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99990c52528042d783bfc0b3815c0715 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:01:57.416285 18859 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99990c52528042d783bfc0b3815c0715 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99990c52528042d783bfc0b3815c0715" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 35249 } }
I20260808 07:01:57.416394 18859 leader_election.cc:302] T 00000000000000000000000000000000 P 99990c52528042d783bfc0b3815c0715 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99990c52528042d783bfc0b3815c0715; no voters:
I20260808 07:01:57.416563 18859 leader_election.cc:288] T 00000000000000000000000000000000 P 99990c52528042d783bfc0b3815c0715 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:01:57.416635 18864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99990c52528042d783bfc0b3815c0715 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:01:57.416811 18864 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99990c52528042d783bfc0b3815c0715 [term 1 LEADER]: Becoming Leader. State: Replica: 99990c52528042d783bfc0b3815c0715, State: Running, Role: LEADER
I20260808 07:01:57.416929 18859 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99990c52528042d783bfc0b3815c0715 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:01:57.416952 18864 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99990c52528042d783bfc0b3815c0715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99990c52528042d783bfc0b3815c0715" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 35249 } }
I20260808 07:01:57.417351 18866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99990c52528042d783bfc0b3815c0715 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99990c52528042d783bfc0b3815c0715. Latest consensus state: current_term: 1 leader_uuid: "99990c52528042d783bfc0b3815c0715" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99990c52528042d783bfc0b3815c0715" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 35249 } } }
I20260808 07:01:57.417346 18865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99990c52528042d783bfc0b3815c0715 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99990c52528042d783bfc0b3815c0715" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99990c52528042d783bfc0b3815c0715" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 35249 } } }
I20260808 07:01:57.417462 18866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99990c52528042d783bfc0b3815c0715 [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:57.417466 18865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99990c52528042d783bfc0b3815c0715 [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:57.417970 18873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:01:57.418483 18873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:01:57.420028 18873 catalog_manager.cc:1347] Generated new cluster ID: 295307e892d74cd39c06748a03583260
I20260808 07:01:57.420083 18873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:01:57.436280 18873 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:01:57.436818 18873 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:01:57.443037 18873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99990c52528042d783bfc0b3815c0715: Generated new TSK 0
I20260808 07:01:57.443229 18873 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:01:57.450856 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.129:0
--local_ip_for_outbound_sockets=127.15.250.129
--webserver_interface=127.15.250.129
--webserver_port=0
--tserver_master_addrs=127.15.250.190:35249
--builtin_ntp_servers=127.15.250.148:33287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000
--consensus_inject_latency_ms_in_notifications=50 with env {}
W20260808 07:01:57.535367 18886 flags.cc:425] Enabled unsafe flag: --consensus_inject_latency_ms_in_notifications=50
W20260808 07:01:57.535596 18886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:57.535612 18886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:57.535643 18886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:57.537048 18886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:57.537117 18886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.129
I20260808 07:01:57.538544 18886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:33287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_inject_latency_ms_in_notifications=50
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.250.129
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.15.250.190:35249
--never_fsync=true
--heap_profile_path=/tmp/kudu.18886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.129
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:01:57.538820 18886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:57.539152 18886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:57.544493 18892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:57.544446 18893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:57.544458 18895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:57.544798 18886 server_base.cc:1034] running on GCE node
I20260808 07:01:57.545027 18886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:57.545295 18886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:57.546474 18886 hybrid_clock.cc:648] HybridClock initialized: now 1786172517546445 us; error 43 us; skew 500 ppm
I20260808 07:01:57.547813 18886 webserver.cc:460] Webserver started at http://127.15.250.129:33587/ using document root <none> and password file <none>
I20260808 07:01:57.548077 18886 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:57.548128 18886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:57.548310 18886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:57.549379 18886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "6ccea5e821f544dda648f4daccee9bc1"
format_stamp: "Formatted at 2026-08-08 07:01:57 on dist-test-slave-0b58"
I20260808 07:01:57.549747 18886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "6ccea5e821f544dda648f4daccee9bc1"
format_stamp: "Formatted at 2026-08-08 07:01:57 on dist-test-slave-0b58"
I20260808 07:01:57.551154 18886 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 07:01:57.552129 18901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:57.552357 18886 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 07:01:57.552474 18886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6ccea5e821f544dda648f4daccee9bc1"
format_stamp: "Formatted at 2026-08-08 07:01:57 on dist-test-slave-0b58"
I20260808 07:01:57.552564 18886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:57.560669 18886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:57.560956 18886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:57.561095 18886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:57.561318 18886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:01:57.561628 18886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:01:57.561678 18886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:57.561710 18886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:01:57.561736 18886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:57.567606 18886 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.129:46595
I20260808 07:01:57.567652 19021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.129:46595 every 8 connection(s)
I20260808 07:01:57.567976 18886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260808 07:01:57.573968 19022 heartbeater.cc:347] Connected to a master server at 127.15.250.190:35249
I20260808 07:01:57.574093 19022 heartbeater.cc:464] Registering TS with master...
I20260808 07:01:57.574283 19022 heartbeater.cc:511] Master 127.15.250.190:35249 requested a full tablet report, sending...
I20260808 07:01:57.574878 18820 ts_manager.cc:194] Registered new tserver with Master: 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:01:57.575543 18820 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.129:56641
I20260808 07:01:57.576962 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 18886
I20260808 07:01:57.577054 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260808 07:01:57.578032 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.130:0
--local_ip_for_outbound_sockets=127.15.250.130
--webserver_interface=127.15.250.130
--webserver_port=0
--tserver_master_addrs=127.15.250.190:35249
--builtin_ntp_servers=127.15.250.148:33287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000
--consensus_inject_latency_ms_in_notifications=50 with env {}
W20260808 07:01:57.662415 19025 flags.cc:425] Enabled unsafe flag: --consensus_inject_latency_ms_in_notifications=50
W20260808 07:01:57.662676 19025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:57.662703 19025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:57.662765 19025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:57.664045 19025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:57.664137 19025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.130
I20260808 07:01:57.665673 19025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:33287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_inject_latency_ms_in_notifications=50
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.250.130
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.15.250.190:35249
--never_fsync=true
--heap_profile_path=/tmp/kudu.19025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.130
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:01:57.665964 19025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:57.666311 19025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:57.671772 19031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:57.671808 19025 server_base.cc:1034] running on GCE node
W20260808 07:01:57.671808 19030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:57.671783 19034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:57.672401 19025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:57.672710 19025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:57.673899 19025 hybrid_clock.cc:648] HybridClock initialized: now 1786172517673883 us; error 30 us; skew 500 ppm
I20260808 07:01:57.675369 19025 webserver.cc:460] Webserver started at http://127.15.250.130:46745/ using document root <none> and password file <none>
I20260808 07:01:57.675681 19025 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:57.675756 19025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:57.675913 19025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:57.677101 19025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf"
format_stamp: "Formatted at 2026-08-08 07:01:57 on dist-test-slave-0b58"
I20260808 07:01:57.677485 19025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf"
format_stamp: "Formatted at 2026-08-08 07:01:57 on dist-test-slave-0b58"
I20260808 07:01:57.678887 19025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 07:01:57.679925 19040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:57.680200 19025 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:01:57.680344 19025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf"
format_stamp: "Formatted at 2026-08-08 07:01:57 on dist-test-slave-0b58"
I20260808 07:01:57.680424 19025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:57.693681 19025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:57.694043 19025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:57.694274 19025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:57.694532 19025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:01:57.694888 19025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:01:57.694944 19025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:57.694986 19025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:01:57.695044 19025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:57.701110 19025 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.130:36995
I20260808 07:01:57.701189 19159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.130:36995 every 8 connection(s)
I20260808 07:01:57.701558 19025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260808 07:01:57.703642 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 19025
I20260808 07:01:57.703759 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260808 07:01:57.705090 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.131:0
--local_ip_for_outbound_sockets=127.15.250.131
--webserver_interface=127.15.250.131
--webserver_port=0
--tserver_master_addrs=127.15.250.190:35249
--builtin_ntp_servers=127.15.250.148:33287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000
--consensus_inject_latency_ms_in_notifications=50 with env {}
I20260808 07:01:57.708619 19160 heartbeater.cc:347] Connected to a master server at 127.15.250.190:35249
I20260808 07:01:57.708737 19160 heartbeater.cc:464] Registering TS with master...
I20260808 07:01:57.708962 19160 heartbeater.cc:511] Master 127.15.250.190:35249 requested a full tablet report, sending...
I20260808 07:01:57.709435 18820 ts_manager.cc:194] Registered new tserver with Master: 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:01:57.709829 18820 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.130:56863
W20260808 07:01:57.789165 19164 flags.cc:425] Enabled unsafe flag: --consensus_inject_latency_ms_in_notifications=50
W20260808 07:01:57.789391 19164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:57.789410 19164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:57.789431 19164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:57.790655 19164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:57.790714 19164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.131
I20260808 07:01:57.792104 19164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:33287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_inject_latency_ms_in_notifications=50
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.250.131
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.15.250.190:35249
--never_fsync=true
--heap_profile_path=/tmp/kudu.19164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.131
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:01:57.792376 19164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:57.792698 19164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:57.797909 19172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:57.797972 19164 server_base.cc:1034] running on GCE node
W20260808 07:01:57.797923 19174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:57.797935 19171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:57.798406 19164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:57.798627 19164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:57.799811 19164 hybrid_clock.cc:648] HybridClock initialized: now 1786172517799805 us; error 36 us; skew 500 ppm
I20260808 07:01:57.801322 19164 webserver.cc:460] Webserver started at http://127.15.250.131:40255/ using document root <none> and password file <none>
I20260808 07:01:57.801568 19164 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:57.801635 19164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:57.801786 19164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:57.802793 19164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "220fdc39470a40e0a6aa355385381500"
format_stamp: "Formatted at 2026-08-08 07:01:57 on dist-test-slave-0b58"
I20260808 07:01:57.803174 19164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "220fdc39470a40e0a6aa355385381500"
format_stamp: "Formatted at 2026-08-08 07:01:57 on dist-test-slave-0b58"
I20260808 07:01:57.804600 19164 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 07:01:57.805328 19180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:57.805501 19164 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:01:57.805604 19164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "220fdc39470a40e0a6aa355385381500"
format_stamp: "Formatted at 2026-08-08 07:01:57 on dist-test-slave-0b58"
I20260808 07:01:57.805677 19164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:57.840134 19164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:57.840546 19164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:57.840713 19164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:57.840956 19164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:01:57.841301 19164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:01:57.841354 19164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:57.841398 19164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:01:57.841424 19164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:57.847155 19164 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.131:44619
I20260808 07:01:57.847198 19301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.131:44619 every 8 connection(s)
I20260808 07:01:57.847545 19164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260808 07:01:57.851851 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 19164
I20260808 07:01:57.851979 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260808 07:01:57.853718 19302 heartbeater.cc:347] Connected to a master server at 127.15.250.190:35249
I20260808 07:01:57.853821 19302 heartbeater.cc:464] Registering TS with master...
I20260808 07:01:57.853967 19302 heartbeater.cc:511] Master 127.15.250.190:35249 requested a full tablet report, sending...
I20260808 07:01:57.854300 18820 ts_manager.cc:194] Registered new tserver with Master: 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619)
I20260808 07:01:57.854700 18820 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.131:49087
I20260808 07:01:57.864455 16362 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 07:01:57.865542 16362 test_util.cc:268] Using random seed: -1301325758
I20260808 07:01:57.872915 18820 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57746:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 07:01:57.873342 18820 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 07:01:57.880807 19229 tablet_service.cc:1467] Processing CreateTablet for tablet 3867cda74ea345ef86c68d815680f9dd (DEFAULT_TABLE table=test-workload [id=3297866f91a041eaa0f9823439545d32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:01:57.880782 19089 tablet_service.cc:1467] Processing CreateTablet for tablet 3867cda74ea345ef86c68d815680f9dd (DEFAULT_TABLE table=test-workload [id=3297866f91a041eaa0f9823439545d32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:01:57.880861 18951 tablet_service.cc:1467] Processing CreateTablet for tablet 3867cda74ea345ef86c68d815680f9dd (DEFAULT_TABLE table=test-workload [id=3297866f91a041eaa0f9823439545d32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:01:57.881201 19229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3867cda74ea345ef86c68d815680f9dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:57.881201 18951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3867cda74ea345ef86c68d815680f9dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:57.881210 19089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3867cda74ea345ef86c68d815680f9dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:57.883788 19317 tablet_bootstrap.cc:492] T 3867cda74ea345ef86c68d815680f9dd P 220fdc39470a40e0a6aa355385381500: Bootstrap starting.
I20260808 07:01:57.884472 19318 tablet_bootstrap.cc:492] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1: Bootstrap starting.
I20260808 07:01:57.884595 19317 tablet_bootstrap.cc:654] T 3867cda74ea345ef86c68d815680f9dd P 220fdc39470a40e0a6aa355385381500: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:57.884727 19319 tablet_bootstrap.cc:492] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf: Bootstrap starting.
I20260808 07:01:57.884946 19317 log.cc:824] T 3867cda74ea345ef86c68d815680f9dd P 220fdc39470a40e0a6aa355385381500: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:57.885253 19318 tablet_bootstrap.cc:654] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:57.885393 19319 tablet_bootstrap.cc:654] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:57.885591 19318 log.cc:824] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:57.885730 19317 tablet_bootstrap.cc:492] T 3867cda74ea345ef86c68d815680f9dd P 220fdc39470a40e0a6aa355385381500: No bootstrap required, opened a new log
I20260808 07:01:57.885741 19319 log.cc:824] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:57.885808 19317 ts_tablet_manager.cc:1397] T 3867cda74ea345ef86c68d815680f9dd P 220fdc39470a40e0a6aa355385381500: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 07:01:57.886384 19319 tablet_bootstrap.cc:492] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf: No bootstrap required, opened a new log
I20260808 07:01:57.886461 19319 ts_tablet_manager.cc:1397] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:01:57.887866 19317 raft_consensus.cc:348] T 3867cda74ea345ef86c68d815680f9dd P 220fdc39470a40e0a6aa355385381500 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:01:57.888002 19317 raft_consensus.cc:374] T 3867cda74ea345ef86c68d815680f9dd P 220fdc39470a40e0a6aa355385381500 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:01:57.888024 19317 raft_consensus.cc:729] T 3867cda74ea345ef86c68d815680f9dd P 220fdc39470a40e0a6aa355385381500 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 220fdc39470a40e0a6aa355385381500, State: Initialized, Role: FOLLOWER
I20260808 07:01:57.888204 19317 consensus_queue.cc:260] T 3867cda74ea345ef86c68d815680f9dd P 220fdc39470a40e0a6aa355385381500 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:01:57.888355 19323 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 220fdc39470a40e0a6aa355385381500 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 07:01:57.888434 19323 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 220fdc39470a40e0a6aa355385381500 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:01:57.888517 19317 ts_tablet_manager.cc:1428] T 3867cda74ea345ef86c68d815680f9dd P 220fdc39470a40e0a6aa355385381500: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 07:01:57.888629 19302 heartbeater.cc:503] Master 127.15.250.190:35249 was elected leader, sending a full tablet report...
I20260808 07:01:57.888808 19323 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 220fdc39470a40e0a6aa355385381500 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:01:57.889245 19318 tablet_bootstrap.cc:492] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1: No bootstrap required, opened a new log
I20260808 07:01:57.889321 19318 ts_tablet_manager.cc:1397] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260808 07:01:57.889766 19319 raft_consensus.cc:348] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:01:57.889936 19319 raft_consensus.cc:374] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:01:57.889969 19319 raft_consensus.cc:729] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fbba1bb0f04471ba1cfa1a42b8619cf, State: Initialized, Role: FOLLOWER
I20260808 07:01:57.890101 19319 consensus_queue.cc:260] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:01:57.890383 19319 ts_tablet_manager.cc:1428] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260808 07:01:57.890406 19160 heartbeater.cc:503] Master 127.15.250.190:35249 was elected leader, sending a full tablet report...
I20260808 07:01:57.891474 19326 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 07:01:57.892108 19318 raft_consensus.cc:348] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:01:57.892249 19326 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:01:57.892316 19318 raft_consensus.cc:374] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:01:57.892354 19318 raft_consensus.cc:729] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ccea5e821f544dda648f4daccee9bc1, State: Initialized, Role: FOLLOWER
I20260808 07:01:57.892525 19318 consensus_queue.cc:260] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:01:57.892565 19111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "220fdc39470a40e0a6aa355385381500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:01:57.892680 19111 raft_consensus.cc:2454] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 220fdc39470a40e0a6aa355385381500 in term 0.
I20260808 07:01:57.892818 19318 ts_tablet_manager.cc:1428] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260808 07:01:57.892941 19022 heartbeater.cc:503] Master 127.15.250.190:35249 was elected leader, sending a full tablet report...
I20260808 07:01:57.892954 19184 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 220fdc39470a40e0a6aa355385381500 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters:
I20260808 07:01:57.893466 19328 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 07:01:57.893601 19328 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:01:57.894361 19328 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:01:57.895500 18973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "220fdc39470a40e0a6aa355385381500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:01:57.895586 18973 raft_consensus.cc:2454] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 220fdc39470a40e0a6aa355385381500 in term 0.
I20260808 07:01:57.895341 19324 raft_consensus.cc:2781] T 3867cda74ea345ef86c68d815680f9dd P 220fdc39470a40e0a6aa355385381500 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 07:01:57.895740 19324 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 220fdc39470a40e0a6aa355385381500 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 07:01:57.895779 19324 raft_consensus.cc:3037] T 3867cda74ea345ef86c68d815680f9dd P 220fdc39470a40e0a6aa355385381500 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:01:57.896440 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:01:57.897392 19324 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 220fdc39470a40e0a6aa355385381500 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:01:57.897531 19324 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 220fdc39470a40e0a6aa355385381500 [CANDIDATE]: Term 1 election: Requested vote from peers 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:01:57.897588 19326 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:01:57.897729 19111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "220fdc39470a40e0a6aa355385381500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf"
I20260808 07:01:57.897807 19111 raft_consensus.cc:3037] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:01:57.898218 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:01:57.898516 19111 raft_consensus.cc:2454] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 220fdc39470a40e0a6aa355385381500 in term 1.
I20260808 07:01:57.897533 19251 raft_consensus.cc:2379] T 3867cda74ea345ef86c68d815680f9dd P 220fdc39470a40e0a6aa355385381500 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6fbba1bb0f04471ba1cfa1a42b8619cf in current term 1: Already voted for candidate 220fdc39470a40e0a6aa355385381500 in this term.
I20260808 07:01:57.898669 19184 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 220fdc39470a40e0a6aa355385381500 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters:
I20260808 07:01:57.898782 19324 raft_consensus.cc:2781] T 3867cda74ea345ef86c68d815680f9dd P 220fdc39470a40e0a6aa355385381500 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:01:57.898870 19324 raft_consensus.cc:686] T 3867cda74ea345ef86c68d815680f9dd P 220fdc39470a40e0a6aa355385381500 [term 1 LEADER]: Becoming Leader. State: Replica: 220fdc39470a40e0a6aa355385381500, State: Running, Role: LEADER
I20260808 07:01:57.898962 19324 consensus_queue.cc:237] T 3867cda74ea345ef86c68d815680f9dd P 220fdc39470a40e0a6aa355385381500 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:01:57.899240 18973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "220fdc39470a40e0a6aa355385381500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1"
W20260808 07:01:57.899256 19041 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.15.250.131:44619 timed out after 0.006s (SENT)
I20260808 07:01:57.899320 18973 raft_consensus.cc:3037] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:01:57.899791 18820 catalog_manager.cc:5495] T 3867cda74ea345ef86c68d815680f9dd P 220fdc39470a40e0a6aa355385381500 reported cstate change: term changed from 0 to 1, leader changed from <none> to 220fdc39470a40e0a6aa355385381500 (127.15.250.131). New cstate: current_term: 1 leader_uuid: "220fdc39470a40e0a6aa355385381500" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } health_report { overall_health: UNKNOWN } } }
I20260808 07:01:57.900136 18973 raft_consensus.cc:2454] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 220fdc39470a40e0a6aa355385381500 in term 1.
W20260808 07:01:57.900645 18902 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.15.250.131:44619 for RPC RequestConsensusVote timed out after 0.006s (ON_OUTBOUND_QUEUE)
I20260808 07:01:57.901327 19324 consensus_queue.cc:1035] T 3867cda74ea345ef86c68d815680f9dd P 220fdc39470a40e0a6aa355385381500 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:01:57.901700 19324 consensus_queue.cc:1035] T 3867cda74ea345ef86c68d815680f9dd P 220fdc39470a40e0a6aa355385381500 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:01:57.902230 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:01:57.902436 18902 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
I20260808 07:01:57.902624 19327 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 07:01:57.902822 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:01:57.902994 18972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:01:57.903218 19043 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters: 220fdc39470a40e0a6aa355385381500, 6ccea5e821f544dda648f4daccee9bc1
I20260808 07:01:57.903332 19326 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 07:01:57.903398 19324 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260808 07:01:57.903437 19324 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20260808 07:01:58.059461 18930 tablet_service.cc:2970] error setting up scanner b464122213b74c9b90479329ab35ae94 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.149s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "fb12f8b5e19d4b6ca5b7af5f7e1f7574"
W20260808 07:01:58.059491 18931 tablet_service.cc:2970] error setting up scanner ae2ee03b222647fd86cd58358b89e16c with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.149s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "dbf78ad7ad494588bbdc886867884ff8"
W20260808 07:01:58.072988 18931 tablet_service.cc:2970] error setting up scanner 7e4150ddb55e424391b5c6822db06243 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.162s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "fb12f8b5e19d4b6ca5b7af5f7e1f7574"
W20260808 07:01:58.075928 18931 tablet_service.cc:2970] error setting up scanner 108d3f1d45dc463ca0af6b6b6dadabd7 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.165s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "dbf78ad7ad494588bbdc886867884ff8"
W20260808 07:01:58.097777 18931 tablet_service.cc:2970] error setting up scanner 68b6673516e44a4cbbc09b75e395bf7a with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.187s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "fb12f8b5e19d4b6ca5b7af5f7e1f7574"
W20260808 07:01:58.098085 19304 tablet.cc:2369] T 3867cda74ea345ef86c68d815680f9dd P 220fdc39470a40e0a6aa355385381500: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 07:01:58.108734 18931 tablet_service.cc:2970] error setting up scanner b9434eefc1644962a02e4714e9544d18 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.198s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "dbf78ad7ad494588bbdc886867884ff8"
W20260808 07:01:58.153630 18931 tablet_service.cc:2970] error setting up scanner 5b2eb5000c5747b7980c26dba33fbc89 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.243s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "dbf78ad7ad494588bbdc886867884ff8"
W20260808 07:01:58.174672 18931 tablet_service.cc:2970] error setting up scanner 5026d242bd544f84b65590fb3dc5bb47 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.264s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "fb12f8b5e19d4b6ca5b7af5f7e1f7574"
W20260808 07:01:58.242971 18973 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260808 07:01:58.242945 19111 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260808 07:01:58.243065 18973 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20260808 07:01:58.243064 19111 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20260808 07:01:58.271752 18931 tablet_service.cc:2970] error setting up scanner 8775c88f0b0847e39ae25695329dcdb0 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.361s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "fb12f8b5e19d4b6ca5b7af5f7e1f7574"
W20260808 07:01:58.298727 18931 tablet_service.cc:2970] error setting up scanner 1103f9f6c93c4dfd9b58c751777a64f9 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.388s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "dbf78ad7ad494588bbdc886867884ff8"
W20260808 07:01:58.576778 18931 tablet_service.cc:2970] error setting up scanner 0b37364e306e4e7188c9b4ac1c58ccf3 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.666s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "fb12f8b5e19d4b6ca5b7af5f7e1f7574"
W20260808 07:01:58.587713 18931 tablet_service.cc:2970] error setting up scanner 3bd0345c9e9c4f57a1d23d03de5be0b5 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.677s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "dbf78ad7ad494588bbdc886867884ff8"
I20260808 07:01:58.815868 19327 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:01:58.815987 19327 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:01:58.816506 19327 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:01:58.816730 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:01:58.816833 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:01:58.817131 18905 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
I20260808 07:01:58.817266 19327 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 07:01:58.961745 18931 tablet_service.cc:2970] error setting up scanner e1e243d943384733a59af4cd3faba712 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.145s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "fb12f8b5e19d4b6ca5b7af5f7e1f7574"
I20260808 07:01:59.057904 19326 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:01:59.058015 19326 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:01:59.058405 19326 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:01:59.058547 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:01:59.058851 18973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:01:59.059055 19043 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters: 220fdc39470a40e0a6aa355385381500, 6ccea5e821f544dda648f4daccee9bc1
I20260808 07:01:59.059211 19326 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 07:01:59.100581 18931 tablet_service.cc:2970] error setting up scanner 5e776260a1b14ffdbff568c174ab4ed2 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.284s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "dbf78ad7ad494588bbdc886867884ff8"
I20260808 07:01:59.203886 19347 mvcc.cc:205] Tried to move back new op lower bound from 7316162633360379904 to 7316162633315123200. Current Snapshot: MvccSnapshot[applied={T|T < 7316162633360379904 or (T in {7316162633360379904})}]
I20260808 07:01:59.237659 19327 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:01:59.237798 19327 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:01:59.238008 19327 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:01:59.238230 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:01:59.238286 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:01:59.238502 18902 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
I20260808 07:01:59.238641 19327 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:01:59.240710 19352 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34320 (ReqId={client: 013957c0297c4f338ebd07fa92c231c7, seq_no=1, attempt_no=0}) took 1331 ms. Trace:
I20260808 07:01:59.240832 19352 rpcz_store.cc:276] 0808 07:01:57.909076 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:57.909095 (+ 19us) service_pool.cc:227] Handling call
0808 07:01:59.240699 (+1331604us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:58.242611 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:01:58.242627 (+ 16us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:58.242627 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:01:58.242628 (+ 1us) tablet.cc:660] Decoding operations
0808 07:01:58.242642 (+ 14us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:58.242651 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:58.242652 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:58.242663 (+ 11us) tablet.cc:699] Row locks acquired
0808 07:01:58.242663 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:01:58.242672 (+ 9us) write_op.cc:270] Start()
0808 07:01:59.203765 (+961093us) write_op.cc:276] Timestamp: P: 1786172518242636 usec, L: 0
0808 07:01:59.203766 (+ 1us) op_driver.cc:331] REPLICATION: starting
0808 07:01:59.203813 (+ 47us) log.cc:842] Serialized 345 byte log entry
0808 07:01:59.240456 (+ 36643us) op_driver.cc:444] REPLICATION: finished
0808 07:01:59.240553 (+ 97us) write_op.cc:301] APPLY: starting
0808 07:01:59.240556 (+ 3us) tablet.cc:1365] starting BulkCheckPresence
0808 07:01:59.240589 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0808 07:01:59.240596 (+ 7us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:01:59.240609 (+ 13us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:01:59.240613 (+ 4us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:01:59.240613 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:01:59.240631 (+ 18us) log.cc:842] Serialized 22 byte log entry
0808 07:01:59.240673 (+ 42us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:01:59.240676 (+ 3us) write_op.cc:454] Released schema lock
0808 07:01:59.240676 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":107008,"child_traces":[["op",{"apply.queue_time_us":77,"mutex_wait_us":2,"num_ops":1,"prepare.queue_time_us":332734,"prepare.run_cpu_time_us":187,"prepare.run_wall_time_us":961251,"raft.queue_time_us":57,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":101,"replication_time_us":36686,"spinlock_wait_cycles":3968}]]}
I20260808 07:01:59.291495 19326 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:01:59.291610 19326 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:01:59.291805 19326 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:01:59.291965 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:01:59.292042 18973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:01:59.292316 19043 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters: 220fdc39470a40e0a6aa355385381500, 6ccea5e821f544dda648f4daccee9bc1
I20260808 07:01:59.292434 19326 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 07:01:59.986855 18931 tablet_service.cc:2970] error setting up scanner fe19427c597e46b897fc85a7f1de87a5 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.748s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "fb12f8b5e19d4b6ca5b7af5f7e1f7574"
W20260808 07:02:00.253691 18931 tablet_service.cc:2970] error setting up scanner 50b1203bcc3641c7b44a20f16f8f413a with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 1.015s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "dbf78ad7ad494588bbdc886867884ff8"
I20260808 07:02:00.254935 19326 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:00.255015 19326 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:00.255223 19326 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:02:00.255450 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:00.255493 18972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:02:00.255772 19043 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters: 220fdc39470a40e0a6aa355385381500, 6ccea5e821f544dda648f4daccee9bc1
I20260808 07:02:00.255911 19326 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:00.289007 19362 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34320 (ReqId={client: 013957c0297c4f338ebd07fa92c231c7, seq_no=2, attempt_no=0}) took 1430 ms. Trace:
I20260808 07:02:00.289103 19362 rpcz_store.cc:276] 0808 07:01:58.858675 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:58.858713 (+ 38us) service_pool.cc:227] Handling call
0808 07:02:00.288990 (+1430277us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:59.204029 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:01:59.204043 (+ 14us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:59.204043 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:01:59.204044 (+ 1us) tablet.cc:660] Decoding operations
0808 07:01:59.204058 (+ 14us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:59.204060 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:59.204061 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:59.204072 (+ 11us) tablet.cc:699] Row locks acquired
0808 07:01:59.204072 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:01:59.204079 (+ 7us) write_op.cc:270] Start()
0808 07:01:59.753211 (+549132us) write_op.cc:276] Timestamp: P: 1786172519204036 usec, L: 0
0808 07:01:59.753213 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:01:59.753260 (+ 47us) log.cc:842] Serialized 354 byte log entry
0808 07:02:00.288718 (+535458us) op_driver.cc:444] REPLICATION: finished
0808 07:02:00.288864 (+ 146us) write_op.cc:301] APPLY: starting
0808 07:02:00.288869 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:00.288874 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:00.288877 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:00.288893 (+ 16us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:00.288897 (+ 4us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:00.288897 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:02:00.288921 (+ 24us) log.cc:842] Serialized 22 byte log entry
0808 07:02:00.288957 (+ 36us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:00.288961 (+ 4us) write_op.cc:454] Released schema lock
0808 07:02:00.288961 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":129,"num_ops":1,"prepare.queue_time_us":345200,"prepare.run_cpu_time_us":191,"prepare.run_wall_time_us":549291,"raft.queue_time_us":84,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":120,"replication_time_us":535499,"spinlock_wait_cycles":19968,"thread_start_us":70,"threads_started":1}]]}
I20260808 07:02:00.624049 19357 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:00.624250 19357 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:00.624490 19357 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:02:00.624689 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:00.624735 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:02:00.625034 18905 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
I20260808 07:02:01.002271 19357 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:01.026222 19367 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34320 (ReqId={client: 013957c0297c4f338ebd07fa92c231c7, seq_no=3, attempt_no=0}) took 1784 ms. Trace:
I20260808 07:02:01.026331 19367 rpcz_store.cc:276] 0808 07:01:59.241519 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:01:59.241552 (+ 33us) service_pool.cc:227] Handling call
0808 07:02:01.026183 (+1784631us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:01:59.753331 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:01:59.753347 (+ 16us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:01:59.753348 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:01:59.753348 (+ 0us) tablet.cc:660] Decoding operations
0808 07:01:59.753362 (+ 14us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:01:59.753365 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:01:59.753366 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:01:59.753378 (+ 12us) tablet.cc:699] Row locks acquired
0808 07:01:59.753378 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:01:59.753386 (+ 8us) write_op.cc:270] Start()
0808 07:01:59.796498 (+ 43112us) write_op.cc:276] Timestamp: P: 1786172519753334 usec, L: 0
0808 07:01:59.796500 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:01:59.796579 (+ 79us) log.cc:842] Serialized 354 byte log entry
0808 07:02:01.025920 (+1229341us) op_driver.cc:444] REPLICATION: finished
0808 07:02:01.026051 (+ 131us) write_op.cc:301] APPLY: starting
0808 07:02:01.026056 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:01.026061 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:01.026064 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:01.026079 (+ 15us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:01.026085 (+ 6us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:01.026086 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:01.026111 (+ 25us) log.cc:842] Serialized 22 byte log entry
0808 07:02:01.026155 (+ 44us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:01.026159 (+ 4us) write_op.cc:454] Released schema lock
0808 07:02:01.026159 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":108,"num_ops":1,"prepare.queue_time_us":511649,"prepare.run_cpu_time_us":210,"prepare.run_wall_time_us":43276,"replication_time_us":1229415,"thread_start_us":80,"threads_started":1}]]}
I20260808 07:02:01.170497 19326 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:01.170607 19326 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:01.170845 19326 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:02:01.170957 18973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:02:01.170967 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:01.171177 19041 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters: 220fdc39470a40e0a6aa355385381500, 6ccea5e821f544dda648f4daccee9bc1
I20260808 07:02:01.171303 19326 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 07:02:01.651882 18931 tablet_service.cc:2970] error setting up scanner 9052ad10cb4d40c49848b14b6346f6b0 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.649s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "fb12f8b5e19d4b6ca5b7af5f7e1f7574"
I20260808 07:02:02.064108 19326 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:02.064235 19326 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:02.064464 19326 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:02:02.064654 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:02.064747 18972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:02:02.065027 19043 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters: 220fdc39470a40e0a6aa355385381500, 6ccea5e821f544dda648f4daccee9bc1
I20260808 07:02:02.098263 19375 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34320 (ReqId={client: 013957c0297c4f338ebd07fa92c231c7, seq_no=4, attempt_no=0}) took 1808 ms. Trace:
I20260808 07:02:02.098351 19375 rpcz_store.cc:276] 0808 07:02:00.289806 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:00.289844 (+ 38us) service_pool.cc:227] Handling call
0808 07:02:02.098251 (+1808407us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:00.289989 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:00.290000 (+ 11us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:00.290001 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:00.290001 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:00.290012 (+ 11us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:00.290014 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:00.290015 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:00.290023 (+ 8us) tablet.cc:699] Row locks acquired
0808 07:02:00.290023 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:00.290029 (+ 6us) write_op.cc:270] Start()
0808 07:02:01.195124 (+905095us) write_op.cc:276] Timestamp: P: 1786172520289969 usec, L: 0
0808 07:02:01.195126 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:02:01.195186 (+ 60us) log.cc:842] Serialized 354 byte log entry
0808 07:02:02.097873 (+902687us) op_driver.cc:444] REPLICATION: finished
0808 07:02:02.098105 (+ 232us) write_op.cc:301] APPLY: starting
0808 07:02:02.098110 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:02.098116 (+ 6us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:02.098119 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:02.098138 (+ 19us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:02.098144 (+ 6us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:02.098145 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:02.098171 (+ 26us) log.cc:842] Serialized 22 byte log entry
0808 07:02:02.098220 (+ 49us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:02.098223 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:02.098223 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":181,"num_ops":1,"prepare.queue_time_us":29,"prepare.run_cpu_time_us":156,"prepare.run_wall_time_us":905221,"replication_time_us":902741,"spinlock_wait_cycles":1280,"thread_start_us":98,"threads_started":1}]]}
I20260808 07:02:02.117151 19357 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:02.117336 19357 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:02.117656 19357 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:02:02.117837 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:02.117871 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:02:02.118124 18905 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
I20260808 07:02:02.717489 19357 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:02.764518 19385 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34320 (ReqId={client: 013957c0297c4f338ebd07fa92c231c7, seq_no=5, attempt_no=0}) took 1737 ms. Trace:
I20260808 07:02:02.764598 19385 rpcz_store.cc:276] 0808 07:02:01.026991 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:01.027019 (+ 28us) service_pool.cc:227] Handling call
0808 07:02:02.764504 (+1737485us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:01.195217 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:01.195234 (+ 17us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:01.195235 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:01.195236 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:01.195250 (+ 14us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:01.195252 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:01.195254 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:01.195265 (+ 11us) tablet.cc:699] Row locks acquired
0808 07:02:01.195266 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:02:01.195273 (+ 7us) write_op.cc:270] Start()
0808 07:02:01.594402 (+399129us) write_op.cc:276] Timestamp: P: 1786172521195232 usec, L: 0
0808 07:02:01.594404 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:02:01.594467 (+ 63us) log.cc:842] Serialized 354 byte log entry
0808 07:02:02.764283 (+1169816us) op_driver.cc:444] REPLICATION: finished
0808 07:02:02.764414 (+ 131us) write_op.cc:301] APPLY: starting
0808 07:02:02.764418 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:02.764422 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:02.764425 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:02.764439 (+ 14us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:02.764444 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:02.764445 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:02.764465 (+ 20us) log.cc:842] Serialized 22 byte log entry
0808 07:02:02.764482 (+ 17us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:02.764484 (+ 2us) write_op.cc:454] Released schema lock
0808 07:02:02.764485 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":119,"num_ops":1,"prepare.queue_time_us":168058,"prepare.run_cpu_time_us":190,"prepare.run_wall_time_us":399276,"replication_time_us":1169874,"thread_start_us":75,"threads_started":1}]]}
I20260808 07:02:02.804513 19326 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:03.016304 19326 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:03.016438 19326 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:03.016680 19326 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:02:03.016855 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:03.016920 18972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:02:03.017163 19043 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters: 220fdc39470a40e0a6aa355385381500, 6ccea5e821f544dda648f4daccee9bc1
I20260808 07:02:03.017251 19326 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 07:02:03.174908 18931 tablet_service.cc:2970] error setting up scanner 29c65c1427c44b7a8c9af7be346c2755 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1786172522174684 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1786172521195232 usec, L: 0 Physical time difference: 0.979s: new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "dbf78ad7ad494588bbdc886867884ff8"
I20260808 07:02:03.761559 19357 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:03.761673 19357 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:03.761909 19357 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:02:03.762179 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:02:03.762179 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:03.762465 18902 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
I20260808 07:02:03.771795 19389 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:03.771911 19389 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:03.772229 19389 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:02:03.772450 18972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:02:03.772430 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:03.772742 19041 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters: 220fdc39470a40e0a6aa355385381500, 6ccea5e821f544dda648f4daccee9bc1
I20260808 07:02:03.772869 19389 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:03.797736 19397 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34320 (ReqId={client: 013957c0297c4f338ebd07fa92c231c7, seq_no=7, attempt_no=0}) took 1032 ms. Trace:
I20260808 07:02:03.797920 19397 rpcz_store.cc:276] 0808 07:02:02.765256 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:02.765286 (+ 30us) service_pool.cc:227] Handling call
0808 07:02:03.797723 (+1032437us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:02.940759 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:02.940777 (+ 18us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:02.940778 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:02.940778 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:02.940792 (+ 14us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:02.940795 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:02.940796 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:02.940807 (+ 11us) tablet.cc:699] Row locks acquired
0808 07:02:02.940807 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:02.940815 (+ 8us) write_op.cc:270] Start()
0808 07:02:03.723918 (+783103us) write_op.cc:276] Timestamp: P: 1786172522940795 usec, L: 0
0808 07:02:03.723919 (+ 1us) op_driver.cc:331] REPLICATION: starting
0808 07:02:03.723967 (+ 48us) log.cc:842] Serialized 354 byte log entry
0808 07:02:03.797431 (+ 73464us) op_driver.cc:444] REPLICATION: finished
0808 07:02:03.797622 (+ 191us) write_op.cc:301] APPLY: starting
0808 07:02:03.797633 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:03.797637 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:03.797639 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:03.797653 (+ 14us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:03.797657 (+ 4us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:03.797658 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:03.797679 (+ 21us) log.cc:842] Serialized 22 byte log entry
0808 07:02:03.797699 (+ 20us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:03.797702 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:03.797703 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":174,"num_ops":1,"prepare.queue_time_us":175366,"prepare.run_cpu_time_us":165,"prepare.run_wall_time_us":783238,"raft.queue_time_us":35,"raft.run_cpu_time_us":46,"raft.run_wall_time_us":46,"replication_time_us":73507,"thread_start_us":85,"threads_started":1}]]}
I20260808 07:02:04.182734 19357 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:04.864104 19399 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:04.864240 19399 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:04.864478 19399 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:02:04.864679 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:04.864711 18972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:02:04.865017 19043 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters: 220fdc39470a40e0a6aa355385381500, 6ccea5e821f544dda648f4daccee9bc1
I20260808 07:02:04.865141 19399 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:04.890043 19403 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34320 (ReqId={client: 013957c0297c4f338ebd07fa92c231c7, seq_no=8, attempt_no=0}) took 1838 ms. Trace:
I20260808 07:02:04.890141 19403 rpcz_store.cc:276] 0808 07:02:03.051593 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:03.051633 (+ 40us) service_pool.cc:227] Handling call
0808 07:02:04.890027 (+1838394us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:03.724004 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:03.724016 (+ 12us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:03.724017 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:03.724017 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:03.724028 (+ 11us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:03.724031 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:03.724032 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:03.724042 (+ 10us) tablet.cc:699] Row locks acquired
0808 07:02:03.724042 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:03.724048 (+ 6us) write_op.cc:270] Start()
0808 07:02:04.525167 (+801119us) write_op.cc:276] Timestamp: P: 1786172523724004 usec, L: 0
0808 07:02:04.525169 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:02:04.525321 (+ 152us) log.cc:842] Serialized 354 byte log entry
0808 07:02:04.889694 (+364373us) op_driver.cc:444] REPLICATION: finished
0808 07:02:04.889892 (+ 198us) write_op.cc:301] APPLY: starting
0808 07:02:04.889896 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:04.889900 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:04.889902 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:04.889915 (+ 13us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:04.889919 (+ 4us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:04.889920 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:04.889939 (+ 19us) log.cc:842] Serialized 22 byte log entry
0808 07:02:04.890000 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:04.890003 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:04.890004 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":179,"num_ops":1,"prepare.queue_time_us":672265,"prepare.run_cpu_time_us":347,"prepare.run_wall_time_us":801432,"raft.queue_time_us":45,"raft.run_cpu_time_us":2,"raft.run_wall_time_us":1,"replication_time_us":364520,"thread_start_us":209,"threads_started":2}]]}
I20260808 07:02:05.378038 19357 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:05.378340 19357 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:05.378556 19357 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:02:05.378783 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:05.378789 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:02:05.379045 18905 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
I20260808 07:02:05.647279 19357 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:05.693311 19409 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34320 (ReqId={client: 013957c0297c4f338ebd07fa92c231c7, seq_no=9, attempt_no=0}) took 1894 ms. Trace:
I20260808 07:02:05.693467 19409 rpcz_store.cc:276] 0808 07:02:03.798590 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:03.798644 (+ 54us) service_pool.cc:227] Handling call
0808 07:02:05.693299 (+1894655us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:04.525444 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:04.525460 (+ 16us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:04.525461 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:04.525461 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:04.525476 (+ 15us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:04.525479 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:04.525481 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:04.525491 (+ 10us) tablet.cc:699] Row locks acquired
0808 07:02:04.525492 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:02:04.525508 (+ 16us) write_op.cc:270] Start()
0808 07:02:05.318597 (+793089us) write_op.cc:276] Timestamp: P: 1786172524525482 usec, L: 0
0808 07:02:05.318599 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:02:05.318664 (+ 65us) log.cc:842] Serialized 354 byte log entry
0808 07:02:05.693023 (+374359us) op_driver.cc:444] REPLICATION: finished
0808 07:02:05.693176 (+ 153us) write_op.cc:301] APPLY: starting
0808 07:02:05.693181 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:05.693186 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:05.693189 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:05.693208 (+ 19us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:05.693213 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:05.693213 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:02:05.693233 (+ 20us) log.cc:842] Serialized 22 byte log entry
0808 07:02:05.693276 (+ 43us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:05.693279 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:05.693279 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":140,"num_ops":1,"prepare.queue_time_us":726673,"prepare.run_cpu_time_us":200,"prepare.run_wall_time_us":793256,"replication_time_us":374419,"thread_start_us":63,"threads_started":1}]]}
I20260808 07:02:06.031456 19399 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:06.031577 19399 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:06.031804 19399 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:02:06.031948 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:06.031976 18973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:02:06.032308 19043 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters: 220fdc39470a40e0a6aa355385381500, 6ccea5e821f544dda648f4daccee9bc1
I20260808 07:02:06.032405 19399 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 07:02:06.248055 18931 tablet_service.cc:2970] error setting up scanner 3fdffba6871a460da217ea96d6356153 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.600s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "dbf78ad7ad494588bbdc886867884ff8"
I20260808 07:02:06.515727 19399 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:06.515849 19399 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:06.516086 19399 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:02:06.516278 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:06.516343 18972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:02:06.516642 19043 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters: 220fdc39470a40e0a6aa355385381500, 6ccea5e821f544dda648f4daccee9bc1
I20260808 07:02:06.516822 19399 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 07:02:06.516872 18931 tablet_service.cc:2970] error setting up scanner b4c6a9dc4cd54b9baaa04e6b0f5136ba with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.869s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "fb12f8b5e19d4b6ca5b7af5f7e1f7574"
I20260808 07:02:06.547592 19415 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34320 (ReqId={client: 013957c0297c4f338ebd07fa92c231c7, seq_no=10, attempt_no=0}) took 1656 ms. Trace:
I20260808 07:02:06.547701 19415 rpcz_store.cc:276] 0808 07:02:04.890819 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:04.890863 (+ 44us) service_pool.cc:227] Handling call
0808 07:02:06.547577 (+1656714us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:05.318708 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:05.318745 (+ 37us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:05.318746 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:05.318747 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:05.318760 (+ 13us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:05.318762 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:05.318764 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:05.318774 (+ 10us) tablet.cc:699] Row locks acquired
0808 07:02:05.318774 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:05.318783 (+ 9us) write_op.cc:270] Start()
0808 07:02:06.210891 (+892108us) write_op.cc:276] Timestamp: P: 1786172525318741 usec, L: 0
0808 07:02:06.210892 (+ 1us) op_driver.cc:331] REPLICATION: starting
0808 07:02:06.210940 (+ 48us) log.cc:842] Serialized 354 byte log entry
0808 07:02:06.547268 (+336328us) op_driver.cc:444] REPLICATION: finished
0808 07:02:06.547443 (+ 175us) write_op.cc:301] APPLY: starting
0808 07:02:06.547448 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:06.547451 (+ 3us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:06.547454 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:06.547467 (+ 13us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:06.547472 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:06.547472 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:02:06.547492 (+ 20us) log.cc:842] Serialized 22 byte log entry
0808 07:02:06.547548 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:06.547551 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:06.547552 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":162,"num_ops":1,"prepare.queue_time_us":427746,"prepare.run_cpu_time_us":220,"prepare.run_wall_time_us":892295,"raft.queue_time_us":80,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":104,"replication_time_us":336370,"spinlock_wait_cycles":8832,"thread_start_us":66,"threads_started":1}]]}
I20260808 07:02:06.781694 19414 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:06.781834 19414 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:06.782084 19414 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:02:06.782256 19111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:02:06.782316 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:06.782586 18902 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
I20260808 07:02:06.782843 19414 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:06.932068 19399 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:06.932199 19399 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:06.932466 19399 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:02:06.932674 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:06.932696 18972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:02:06.932910 19043 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters: 220fdc39470a40e0a6aa355385381500, 6ccea5e821f544dda648f4daccee9bc1
I20260808 07:02:06.933043 19399 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:06.976140 19415 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34320 (ReqId={client: 013957c0297c4f338ebd07fa92c231c7, seq_no=11, attempt_no=0}) took 1282 ms. Trace:
I20260808 07:02:06.976310 19415 rpcz_store.cc:276] 0808 07:02:05.694017 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:05.694038 (+ 21us) service_pool.cc:227] Handling call
0808 07:02:06.976124 (+1282086us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:06.211016 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:06.211032 (+ 16us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:06.211032 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:06.211033 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:06.211046 (+ 13us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:06.211049 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:06.211050 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:06.211070 (+ 20us) tablet.cc:699] Row locks acquired
0808 07:02:06.211070 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:06.211078 (+ 8us) write_op.cc:270] Start()
0808 07:02:06.823267 (+612189us) write_op.cc:276] Timestamp: P: 1786172526211047 usec, L: 0
0808 07:02:06.823269 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:02:06.823335 (+ 66us) log.cc:842] Serialized 354 byte log entry
0808 07:02:06.975807 (+152472us) op_driver.cc:444] REPLICATION: finished
0808 07:02:06.975884 (+ 77us) write_op.cc:301] APPLY: starting
0808 07:02:06.975889 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:06.975895 (+ 6us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:06.975899 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:06.975915 (+ 16us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:06.975921 (+ 6us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:06.975921 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:02:06.975951 (+ 30us) log.cc:842] Serialized 22 byte log entry
0808 07:02:06.976089 (+ 138us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:06.976093 (+ 4us) write_op.cc:454] Released schema lock
0808 07:02:06.976094 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":56,"num_ops":1,"prepare.queue_time_us":516872,"prepare.run_cpu_time_us":235,"prepare.run_wall_time_us":612384,"raft.queue_time_us":107,"raft.run_cpu_time_us":51,"raft.run_wall_time_us":50,"replication_time_us":152532}]]}
I20260808 07:02:07.410123 19414 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:07.410246 19414 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:07.410487 19414 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:02:07.410619 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:07.410696 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:02:07.410902 18905 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
I20260808 07:02:07.843406 19399 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:07.843532 19399 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:07.843788 19399 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:02:07.844005 18972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:02:07.844043 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:07.844487 19041 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters: 220fdc39470a40e0a6aa355385381500, 6ccea5e821f544dda648f4daccee9bc1
I20260808 07:02:07.861372 19427 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34320 (ReqId={client: 013957c0297c4f338ebd07fa92c231c7, seq_no=12, attempt_no=0}) took 1312 ms. Trace:
I20260808 07:02:07.861486 19427 rpcz_store.cc:276] 0808 07:02:06.548384 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:06.548426 (+ 42us) service_pool.cc:227] Handling call
0808 07:02:07.861358 (+1312932us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:06.823410 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:06.823427 (+ 17us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:06.823427 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:06.823428 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:06.823441 (+ 13us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:06.823444 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:06.823445 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:06.823456 (+ 11us) tablet.cc:699] Row locks acquired
0808 07:02:06.823457 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:02:06.823465 (+ 8us) write_op.cc:270] Start()
0808 07:02:06.937558 (+114093us) write_op.cc:276] Timestamp: P: 1786172526823417 usec, L: 0
0808 07:02:06.937559 (+ 1us) op_driver.cc:331] REPLICATION: starting
0808 07:02:06.937601 (+ 42us) log.cc:842] Serialized 354 byte log entry
0808 07:02:07.860993 (+923392us) op_driver.cc:444] REPLICATION: finished
0808 07:02:07.861249 (+ 256us) write_op.cc:301] APPLY: starting
0808 07:02:07.861254 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:07.861258 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:07.861261 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:07.861276 (+ 15us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:07.861281 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:07.861282 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:07.861305 (+ 23us) log.cc:842] Serialized 22 byte log entry
0808 07:02:07.861329 (+ 24us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:07.861332 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:07.861333 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":239,"num_ops":1,"prepare.queue_time_us":274872,"prepare.run_cpu_time_us":152,"prepare.run_wall_time_us":114220,"raft.queue_time_us":37,"raft.run_cpu_time_us":62,"raft.run_wall_time_us":63,"replication_time_us":923427,"thread_start_us":89,"threads_started":1}]]}
I20260808 07:02:07.950688 19399 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:08.177346 19414 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:08.488291 19414 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:08.488426 19414 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:08.488669 19414 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:02:08.488867 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:08.488869 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:02:08.489159 18905 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
I20260808 07:02:08.489290 19414 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:08.810536 19399 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:08.810650 19399 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:08.810899 19399 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:02:08.811028 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:08.811159 18972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:02:08.811379 19043 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters: 220fdc39470a40e0a6aa355385381500, 6ccea5e821f544dda648f4daccee9bc1
I20260808 07:02:09.456035 19414 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:09.456147 19414 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:09.456379 19414 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:02:09.456564 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:09.456681 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:02:09.457047 18905 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
I20260808 07:02:09.497076 19439 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34320 (ReqId={client: 013957c0297c4f338ebd07fa92c231c7, seq_no=15, attempt_no=0}) took 1526 ms. Trace:
I20260808 07:02:09.497231 19439 rpcz_store.cc:276] 0808 07:02:07.971043 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:07.971092 (+ 49us) service_pool.cc:227] Handling call
0808 07:02:09.497063 (+1525971us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:08.348698 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:08.348711 (+ 13us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:08.348711 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:08.348712 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:08.348722 (+ 10us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:08.348725 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:08.348726 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:08.348735 (+ 9us) tablet.cc:699] Row locks acquired
0808 07:02:08.348736 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:02:08.348742 (+ 6us) write_op.cc:270] Start()
0808 07:02:08.429850 (+ 81108us) write_op.cc:276] Timestamp: P: 1786172528348702 usec, L: 0
0808 07:02:08.429853 (+ 3us) op_driver.cc:331] REPLICATION: starting
0808 07:02:08.429917 (+ 64us) log.cc:842] Serialized 354 byte log entry
0808 07:02:09.496780 (+1066863us) op_driver.cc:444] REPLICATION: finished
0808 07:02:09.496956 (+ 176us) write_op.cc:301] APPLY: starting
0808 07:02:09.496960 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:09.496964 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:09.496968 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:09.496984 (+ 16us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:09.496988 (+ 4us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:09.496988 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:02:09.497013 (+ 25us) log.cc:842] Serialized 22 byte log entry
0808 07:02:09.497035 (+ 22us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:09.497038 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:09.497039 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":153,"num_ops":1,"prepare.queue_time_us":377485,"prepare.run_cpu_time_us":171,"prepare.run_wall_time_us":81243,"replication_time_us":1066920,"thread_start_us":84,"threads_started":1}]]}
I20260808 07:02:09.595839 19414 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:09.602939 19399 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:10.232101 19399 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:10.232194 19399 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:10.232702 18972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:02:10.232801 19399 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:02:10.232854 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:10.233255 19043 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters: 220fdc39470a40e0a6aa355385381500, 6ccea5e821f544dda648f4daccee9bc1
I20260808 07:02:10.282606 19399 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:10.710366 19445 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:10.710490 19445 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:10.710770 19445 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:02:10.710919 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:10.710943 19111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:02:10.711176 18905 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
I20260808 07:02:10.711319 19445 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 07:02:10.857017 18931 tablet_service.cc:2970] error setting up scanner 6768c0ae70fc45648f93624c91b67da9 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.146s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "dbf78ad7ad494588bbdc886867884ff8"
I20260808 07:02:11.240602 19445 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:11.240734 19445 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:11.240986 19445 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:02:11.241212 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:02:11.241160 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:11.241500 18905 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
I20260808 07:02:11.241693 19445 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:11.268510 19453 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:11.268631 19453 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:11.268877 19453 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:02:11.269032 18973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:02:11.269109 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:11.269416 19041 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters: 220fdc39470a40e0a6aa355385381500, 6ccea5e821f544dda648f4daccee9bc1
I20260808 07:02:11.269546 19453 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:11.271521 19460 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34320 (ReqId={client: 013957c0297c4f338ebd07fa92c231c7, seq_no=18, attempt_no=0}) took 1657 ms. Trace:
I20260808 07:02:11.271582 19460 rpcz_store.cc:276] 0808 07:02:09.614227 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:09.614267 (+ 40us) service_pool.cc:227] Handling call
0808 07:02:11.271512 (+1657245us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:10.192452 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:10.192470 (+ 18us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:10.192471 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:10.192471 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:10.192485 (+ 14us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:10.192488 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:10.192489 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:10.192499 (+ 10us) tablet.cc:699] Row locks acquired
0808 07:02:10.192500 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:02:10.192509 (+ 9us) write_op.cc:270] Start()
0808 07:02:11.030611 (+838102us) write_op.cc:276] Timestamp: P: 1786172530192476 usec, L: 0
0808 07:02:11.030613 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:02:11.030664 (+ 51us) log.cc:842] Serialized 354 byte log entry
0808 07:02:11.271241 (+240577us) op_driver.cc:444] REPLICATION: finished
0808 07:02:11.271388 (+ 147us) write_op.cc:301] APPLY: starting
0808 07:02:11.271417 (+ 29us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:11.271421 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:11.271425 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:11.271439 (+ 14us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:11.271444 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:11.271444 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:02:11.271463 (+ 19us) log.cc:842] Serialized 22 byte log entry
0808 07:02:11.271483 (+ 20us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:11.271485 (+ 2us) write_op.cc:454] Released schema lock
0808 07:02:11.271486 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":135,"num_ops":1,"prepare.queue_time_us":578102,"prepare.run_cpu_time_us":169,"prepare.run_wall_time_us":838236,"replication_time_us":240624,"thread_start_us":91,"threads_started":1}]]}
W20260808 07:02:11.381630 18931 tablet_service.cc:2970] error setting up scanner fad84f8243154e379af98712096477f7 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.140s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "fb12f8b5e19d4b6ca5b7af5f7e1f7574"
I20260808 07:02:12.188067 19462 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:12.188339 19462 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:12.189160 19462 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:02:12.189136 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:12.189502 18972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:02:12.189905 19043 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters: 220fdc39470a40e0a6aa355385381500, 6ccea5e821f544dda648f4daccee9bc1
I20260808 07:02:12.201651 19465 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34320 (ReqId={client: 013957c0297c4f338ebd07fa92c231c7, seq_no=19, attempt_no=0}) took 1908 ms. Trace:
I20260808 07:02:12.201857 19465 rpcz_store.cc:276] 0808 07:02:10.293281 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:10.293335 (+ 54us) service_pool.cc:227] Handling call
0808 07:02:12.201624 (+1908289us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:11.030697 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:11.030714 (+ 17us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:11.030714 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:11.030715 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:11.030728 (+ 13us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:11.030730 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:11.030731 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:11.030742 (+ 11us) tablet.cc:699] Row locks acquired
0808 07:02:11.030742 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:11.030750 (+ 8us) write_op.cc:270] Start()
0808 07:02:11.920900 (+890150us) write_op.cc:276] Timestamp: P: 1786172531030702 usec, L: 0
0808 07:02:11.920902 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:02:11.920969 (+ 67us) log.cc:842] Serialized 354 byte log entry
0808 07:02:12.201173 (+280204us) op_driver.cc:444] REPLICATION: finished
0808 07:02:12.201426 (+ 253us) write_op.cc:301] APPLY: starting
0808 07:02:12.201431 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:12.201435 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:12.201445 (+ 10us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:12.201468 (+ 23us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:12.201473 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:12.201474 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:12.201513 (+ 39us) log.cc:842] Serialized 22 byte log entry
0808 07:02:12.201572 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:12.201576 (+ 4us) write_op.cc:454] Released schema lock
0808 07:02:12.201576 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":229,"num_ops":1,"prepare.queue_time_us":737218,"prepare.run_cpu_time_us":233,"prepare.run_wall_time_us":890323,"raft.queue_time_us":79,"raft.run_cpu_time_us":57,"raft.run_wall_time_us":57,"replication_time_us":280258,"thread_start_us":146,"threads_started":1}]]}
I20260808 07:02:12.567036 19461 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:12.567200 19461 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:12.567438 19461 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:02:12.567628 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:12.567654 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:02:12.567900 18902 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
I20260808 07:02:12.856454 19461 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:12.893482 19474 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34320 (ReqId={client: 013957c0297c4f338ebd07fa92c231c7, seq_no=20, attempt_no=0}) took 1621 ms. Trace:
I20260808 07:02:12.893580 19474 rpcz_store.cc:276] 0808 07:02:11.272227 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:11.272249 (+ 22us) service_pool.cc:227] Handling call
0808 07:02:12.893471 (+1621222us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:11.921028 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:11.921045 (+ 17us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:11.921046 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:11.921046 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:11.921060 (+ 14us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:11.921062 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:11.921067 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:11.921078 (+ 11us) tablet.cc:699] Row locks acquired
0808 07:02:11.921078 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:11.921086 (+ 8us) write_op.cc:270] Start()
0808 07:02:11.930173 (+ 9087us) write_op.cc:276] Timestamp: P: 1786172531921017 usec, L: 0
0808 07:02:11.930176 (+ 3us) op_driver.cc:331] REPLICATION: starting
0808 07:02:11.930265 (+ 89us) log.cc:842] Serialized 354 byte log entry
0808 07:02:12.893117 (+962852us) op_driver.cc:444] REPLICATION: finished
0808 07:02:12.893326 (+ 209us) write_op.cc:301] APPLY: starting
0808 07:02:12.893332 (+ 6us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:12.893335 (+ 3us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:12.893338 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:12.893401 (+ 63us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:12.893407 (+ 6us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:12.893408 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:12.893431 (+ 23us) log.cc:842] Serialized 22 byte log entry
0808 07:02:12.893453 (+ 22us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:12.893454 (+ 1us) write_op.cc:454] Released schema lock
0808 07:02:12.893454 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":197,"num_ops":1,"prepare.queue_time_us":648630,"prepare.run_cpu_time_us":239,"prepare.run_wall_time_us":9259,"replication_time_us":962936,"spinlock_wait_cycles":17792,"thread_start_us":168,"threads_started":1}]]}
I20260808 07:02:12.975296 19462 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:13.074265 19461 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:13.074393 19461 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:13.074635 19461 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:02:13.074844 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:13.074991 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:02:13.075297 18905 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
I20260808 07:02:13.075486 19461 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:13.301205 19462 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:13.301308 19462 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:13.301544 19462 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:02:13.301708 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:13.302253 18973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:02:13.302663 19043 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters: 220fdc39470a40e0a6aa355385381500, 6ccea5e821f544dda648f4daccee9bc1
I20260808 07:02:13.901875 19462 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:14.093580 19483 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:14.093691 19483 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:14.093938 19483 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:02:14.094174 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:02:14.094156 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:14.094470 18905 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
I20260808 07:02:14.094596 19483 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:14.094929 19487 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34320 (ReqId={client: 013957c0297c4f338ebd07fa92c231c7, seq_no=22, attempt_no=0}) took 1200 ms. Trace:
I20260808 07:02:14.095026 19487 rpcz_store.cc:276] 0808 07:02:12.893973 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:12.894015 (+ 42us) service_pool.cc:227] Handling call
0808 07:02:14.094893 (+1200878us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:13.040510 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:13.040525 (+ 15us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:13.040526 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:13.040527 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:13.040539 (+ 12us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:13.040542 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:13.040543 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:13.040554 (+ 11us) tablet.cc:699] Row locks acquired
0808 07:02:13.040554 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:13.040563 (+ 9us) write_op.cc:270] Start()
0808 07:02:13.992503 (+951940us) write_op.cc:276] Timestamp: P: 1786172533040522 usec, L: 0
0808 07:02:13.992504 (+ 1us) op_driver.cc:331] REPLICATION: starting
0808 07:02:13.992564 (+ 60us) log.cc:842] Serialized 354 byte log entry
0808 07:02:14.094382 (+101818us) op_driver.cc:444] REPLICATION: finished
0808 07:02:14.094802 (+ 420us) write_op.cc:301] APPLY: starting
0808 07:02:14.094805 (+ 3us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:14.094809 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:14.094812 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:14.094824 (+ 12us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:14.094829 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:14.094829 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:02:14.094847 (+ 18us) log.cc:842] Serialized 22 byte log entry
0808 07:02:14.094868 (+ 21us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:14.094872 (+ 4us) write_op.cc:454] Released schema lock
0808 07:02:14.094872 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":2432,"child_traces":[["op",{"apply.queue_time_us":204,"num_ops":1,"prepare.queue_time_us":146400,"prepare.run_cpu_time_us":412,"prepare.run_wall_time_us":952076,"replication_time_us":101873,"spinlock_wait_cycles":459648,"thread_start_us":168,"threads_started":1}]]}
I20260808 07:02:14.231555 19483 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:14.231639 19483 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:14.231832 19483 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:02:14.232015 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:14.232033 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:02:14.232460 18905 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
I20260808 07:02:14.232605 19483 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:14.640849 19462 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:14.640959 19462 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:14.641183 19462 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:02:14.641312 18973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:02:14.641390 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:14.641712 19041 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters: 220fdc39470a40e0a6aa355385381500, 6ccea5e821f544dda648f4daccee9bc1
I20260808 07:02:14.641832 19462 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 07:02:14.954398 18931 tablet_service.cc:2970] error setting up scanner 1c833a2bf9b54329954810ddf94e140b with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.860s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "dbf78ad7ad494588bbdc886867884ff8"
I20260808 07:02:15.022554 19483 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:15.022688 19483 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:15.022924 19483 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:02:15.023131 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:15.023175 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:02:15.023392 18905 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
I20260808 07:02:15.023533 19483 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:15.061676 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34320 (ReqId={client: 013957c0297c4f338ebd07fa92c231c7, seq_no=24, attempt_no=0}) took 1945 ms. Trace:
I20260808 07:02:15.061753 19495 rpcz_store.cc:276] 0808 07:02:13.116607 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:13.116631 (+ 24us) service_pool.cc:227] Handling call
0808 07:02:15.061660 (+1945029us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:13.992595 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:13.992612 (+ 17us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:13.992613 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:13.992613 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:13.992628 (+ 15us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:13.992631 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:13.992632 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:13.992642 (+ 10us) tablet.cc:699] Row locks acquired
0808 07:02:13.992642 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:13.992651 (+ 9us) write_op.cc:270] Start()
0808 07:02:14.645769 (+653118us) write_op.cc:276] Timestamp: P: 1786172533992614 usec, L: 0
0808 07:02:14.645770 (+ 1us) op_driver.cc:331] REPLICATION: starting
0808 07:02:14.645809 (+ 39us) log.cc:842] Serialized 354 byte log entry
0808 07:02:15.061277 (+415468us) op_driver.cc:444] REPLICATION: finished
0808 07:02:15.061563 (+ 286us) write_op.cc:301] APPLY: starting
0808 07:02:15.061568 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:15.061572 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:15.061575 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:15.061589 (+ 14us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:15.061593 (+ 4us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:15.061593 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:02:15.061613 (+ 20us) log.cc:842] Serialized 22 byte log entry
0808 07:02:15.061630 (+ 17us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:15.061633 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:15.061633 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":267,"num_ops":1,"prepare.queue_time_us":875874,"prepare.run_cpu_time_us":204,"prepare.run_wall_time_us":653276,"raft.queue_time_us":117,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":105,"replication_time_us":415501,"spinlock_wait_cycles":12672,"thread_start_us":99,"threads_started":1}]]}
I20260808 07:02:15.475569 19462 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:15.475688 19462 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:15.475915 19462 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:02:15.476148 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:15.476143 18972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:02:15.476487 19041 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters: 220fdc39470a40e0a6aa355385381500, 6ccea5e821f544dda648f4daccee9bc1
W20260808 07:02:15.734782 18931 tablet_service.cc:2970] error setting up scanner 54ddef1c52e545a2829983f0ad9bcf80 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.711s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "fb12f8b5e19d4b6ca5b7af5f7e1f7574"
I20260808 07:02:16.119091 19483 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:16.119253 19483 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:16.119485 19483 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:02:16.119634 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:16.119737 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:02:16.119966 18905 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
I20260808 07:02:16.152304 19503 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34320 (ReqId={client: 013957c0297c4f338ebd07fa92c231c7, seq_no=25, attempt_no=0}) took 2056 ms. Trace:
I20260808 07:02:16.152423 19503 rpcz_store.cc:276] 0808 07:02:14.095744 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:14.095772 (+ 28us) service_pool.cc:227] Handling call
0808 07:02:16.152286 (+2056514us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:14.645879 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:14.645902 (+ 23us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:14.645903 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:14.645904 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:14.645950 (+ 46us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:14.645952 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:14.645953 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:14.645963 (+ 10us) tablet.cc:699] Row locks acquired
0808 07:02:14.645963 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:14.645971 (+ 8us) write_op.cc:270] Start()
0808 07:02:15.154121 (+508150us) write_op.cc:276] Timestamp: P: 1786172534645919 usec, L: 0
0808 07:02:15.154122 (+ 1us) op_driver.cc:331] REPLICATION: starting
0808 07:02:15.154172 (+ 50us) log.cc:842] Serialized 354 byte log entry
0808 07:02:16.151940 (+997768us) op_driver.cc:444] REPLICATION: finished
0808 07:02:16.152084 (+ 144us) write_op.cc:301] APPLY: starting
0808 07:02:16.152089 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:16.152093 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:16.152097 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:16.152114 (+ 17us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:16.152118 (+ 4us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:16.152119 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:16.152163 (+ 44us) log.cc:842] Serialized 22 byte log entry
0808 07:02:16.152222 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:16.152226 (+ 4us) write_op.cc:454] Released schema lock
0808 07:02:16.152227 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":127,"num_ops":1,"prepare.queue_time_us":549974,"prepare.run_cpu_time_us":249,"prepare.run_wall_time_us":508363,"raft.queue_time_us":83,"raft.run_cpu_time_us":2,"raft.run_wall_time_us":2,"replication_time_us":997812,"thread_start_us":88,"threads_started":1}]]}
I20260808 07:02:16.194067 19462 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:16.214466 19483 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:16.235831 19503 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34320 (ReqId={client: 013957c0297c4f338ebd07fa92c231c7, seq_no=26, attempt_no=0}) took 1173 ms. Trace:
I20260808 07:02:16.235916 19503 rpcz_store.cc:276] 0808 07:02:15.062339 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:15.062362 (+ 23us) service_pool.cc:227] Handling call
0808 07:02:16.235818 (+1173456us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:15.154251 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:15.154291 (+ 40us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:15.154291 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:15.154292 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:15.154307 (+ 15us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:15.154309 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:15.154311 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:15.154321 (+ 10us) tablet.cc:699] Row locks acquired
0808 07:02:15.154322 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:02:15.154330 (+ 8us) write_op.cc:270] Start()
0808 07:02:15.195474 (+ 41144us) write_op.cc:276] Timestamp: P: 1786172535154277 usec, L: 0
0808 07:02:15.195475 (+ 1us) op_driver.cc:331] REPLICATION: starting
0808 07:02:15.195538 (+ 63us) log.cc:842] Serialized 354 byte log entry
0808 07:02:16.235644 (+1040106us) op_driver.cc:444] REPLICATION: finished
0808 07:02:16.235697 (+ 53us) write_op.cc:301] APPLY: starting
0808 07:02:16.235700 (+ 3us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:16.235705 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:16.235707 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:16.235721 (+ 14us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:16.235726 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:16.235727 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:16.235754 (+ 27us) log.cc:842] Serialized 22 byte log entry
0808 07:02:16.235789 (+ 35us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:16.235793 (+ 4us) write_op.cc:454] Released schema lock
0808 07:02:16.235794 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":40,"num_ops":1,"prepare.queue_time_us":91759,"prepare.run_cpu_time_us":220,"prepare.run_wall_time_us":41311,"replication_time_us":1040163}]]}
I20260808 07:02:16.425254 19462 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:16.425370 19462 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:16.425577 19462 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:02:16.425782 18972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:02:16.425797 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:16.426010 19041 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters: 220fdc39470a40e0a6aa355385381500, 6ccea5e821f544dda648f4daccee9bc1
I20260808 07:02:16.426103 19462 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:17.030328 19483 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:17.030453 19483 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:17.030695 19483 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:02:17.030938 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:17.031035 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:02:17.031248 18905 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
I20260808 07:02:17.269706 19510 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:17.269831 19510 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:17.270171 19510 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:02:17.270231 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:17.270401 18972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:02:17.270623 19043 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters: 220fdc39470a40e0a6aa355385381500, 6ccea5e821f544dda648f4daccee9bc1
I20260808 07:02:17.270720 19510 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:17.295686 19514 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34320 (ReqId={client: 013957c0297c4f338ebd07fa92c231c7, seq_no=28, attempt_no=0}) took 1058 ms. Trace:
I20260808 07:02:17.295794 19514 rpcz_store.cc:276] 0808 07:02:16.236739 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:16.236786 (+ 47us) service_pool.cc:227] Handling call
0808 07:02:17.295669 (+1058883us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:16.411606 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:16.411622 (+ 16us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:16.411622 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:16.411623 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:16.411636 (+ 13us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:16.411639 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:16.411640 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:16.411650 (+ 10us) tablet.cc:699] Row locks acquired
0808 07:02:16.411651 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:02:16.411658 (+ 7us) write_op.cc:270] Start()
0808 07:02:17.023754 (+612096us) write_op.cc:276] Timestamp: P: 1786172536411641 usec, L: 0
0808 07:02:17.023757 (+ 3us) op_driver.cc:331] REPLICATION: starting
0808 07:02:17.023804 (+ 47us) log.cc:842] Serialized 354 byte log entry
0808 07:02:17.295336 (+271532us) op_driver.cc:444] REPLICATION: finished
0808 07:02:17.295503 (+ 167us) write_op.cc:301] APPLY: starting
0808 07:02:17.295508 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:17.295511 (+ 3us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:17.295515 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:17.295530 (+ 15us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:17.295568 (+ 38us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:17.295568 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:02:17.295594 (+ 26us) log.cc:842] Serialized 22 byte log entry
0808 07:02:17.295642 (+ 48us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:17.295646 (+ 4us) write_op.cc:454] Released schema lock
0808 07:02:17.295646 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":154,"num_ops":1,"prepare.queue_time_us":174697,"prepare.run_cpu_time_us":179,"prepare.run_wall_time_us":612247,"raft.queue_time_us":61,"raft.run_cpu_time_us":52,"raft.run_wall_time_us":52,"replication_time_us":271574,"thread_start_us":90,"threads_started":1}]]}
I20260808 07:02:17.485358 19483 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 07:02:17.771317 18931 tablet_service.cc:2970] error setting up scanner 2348404e1cf04c13aa31082de5413f0f with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.285s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "dbf78ad7ad494588bbdc886867884ff8"
I20260808 07:02:17.916926 19483 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:17.917054 19483 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:17.917279 19483 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:02:17.917471 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:17.917534 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:02:17.917793 18905 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
I20260808 07:02:17.917888 19483 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:17.927081 19516 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34320 (ReqId={client: 013957c0297c4f338ebd07fa92c231c7, seq_no=29, attempt_no=0}) took 1488 ms. Trace:
I20260808 07:02:17.927170 19516 rpcz_store.cc:276] 0808 07:02:16.438537 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:16.438561 (+ 24us) service_pool.cc:227] Handling call
0808 07:02:17.927064 (+1488503us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:17.023862 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:17.023878 (+ 16us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:17.023879 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:17.023879 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:17.023893 (+ 14us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:17.023896 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:17.023897 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:17.023907 (+ 10us) tablet.cc:699] Row locks acquired
0808 07:02:17.023908 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:02:17.023916 (+ 8us) write_op.cc:270] Start()
0808 07:02:17.882998 (+859082us) write_op.cc:276] Timestamp: P: 1786172537023889 usec, L: 0
0808 07:02:17.882999 (+ 1us) op_driver.cc:331] REPLICATION: starting
0808 07:02:17.883054 (+ 55us) log.cc:842] Serialized 354 byte log entry
0808 07:02:17.926547 (+ 43493us) op_driver.cc:444] REPLICATION: finished
0808 07:02:17.926942 (+ 395us) write_op.cc:301] APPLY: starting
0808 07:02:17.926948 (+ 6us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:17.926952 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:17.926955 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:17.926968 (+ 13us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:17.926974 (+ 6us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:17.926974 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:02:17.926997 (+ 23us) log.cc:842] Serialized 22 byte log entry
0808 07:02:17.927038 (+ 41us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:17.927041 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:17.927042 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":381,"num_ops":1,"prepare.queue_time_us":585158,"prepare.run_cpu_time_us":188,"prepare.run_wall_time_us":859243,"raft.queue_time_us":80,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":98,"replication_time_us":43544,"thread_start_us":63,"threads_started":1}]]}
I20260808 07:02:18.104737 19515 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:18.104907 19515 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:18.105154 19515 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:02:18.105273 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:18.105399 18973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:02:18.105643 19043 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters: 220fdc39470a40e0a6aa355385381500, 6ccea5e821f544dda648f4daccee9bc1
I20260808 07:02:18.105909 19515 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:18.608321 19515 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:18.608440 19515 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:18.608675 19515 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:02:18.608878 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:18.608865 18972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:02:18.609133 19041 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters: 220fdc39470a40e0a6aa355385381500, 6ccea5e821f544dda648f4daccee9bc1
I20260808 07:02:18.609272 19515 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:18.653266 19521 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34320 (ReqId={client: 013957c0297c4f338ebd07fa92c231c7, seq_no=30, attempt_no=0}) took 1356 ms. Trace:
I20260808 07:02:18.653385 19521 rpcz_store.cc:276] 0808 07:02:17.296347 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:17.296390 (+ 43us) service_pool.cc:227] Handling call
0808 07:02:18.653242 (+1356852us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:17.883111 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:17.883127 (+ 16us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:17.883127 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:17.883128 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:17.883141 (+ 13us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:17.883143 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:17.883145 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:17.883155 (+ 10us) tablet.cc:699] Row locks acquired
0808 07:02:17.883156 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:02:17.883164 (+ 8us) write_op.cc:270] Start()
0808 07:02:18.171433 (+288269us) write_op.cc:276] Timestamp: P: 1786172537883083 usec, L: 0
0808 07:02:18.171435 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:02:18.171490 (+ 55us) log.cc:842] Serialized 354 byte log entry
0808 07:02:18.652898 (+481408us) op_driver.cc:444] REPLICATION: finished
0808 07:02:18.653056 (+ 158us) write_op.cc:301] APPLY: starting
0808 07:02:18.653061 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:18.653065 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:18.653068 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:18.653081 (+ 13us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:18.653085 (+ 4us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:18.653086 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:18.653106 (+ 20us) log.cc:842] Serialized 22 byte log entry
0808 07:02:18.653213 (+ 107us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:18.653217 (+ 4us) write_op.cc:454] Released schema lock
0808 07:02:18.653218 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":140,"num_ops":1,"prepare.queue_time_us":586616,"prepare.run_cpu_time_us":327,"prepare.run_wall_time_us":288400,"replication_time_us":481458,"thread_start_us":62,"threads_started":1}]]}
I20260808 07:02:18.933413 19483 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:18.933542 19483 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:18.933881 19483 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:02:18.934047 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:18.934212 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:02:18.934438 18905 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
I20260808 07:02:19.438891 19515 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:19.438994 19515 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:19.439262 19515 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:02:19.439409 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:19.439517 18972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:02:19.439788 19043 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters: 220fdc39470a40e0a6aa355385381500, 6ccea5e821f544dda648f4daccee9bc1
I20260808 07:02:19.469137 19530 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34320 (ReqId={client: 013957c0297c4f338ebd07fa92c231c7, seq_no=31, attempt_no=0}) took 1541 ms. Trace:
I20260808 07:02:19.469269 19530 rpcz_store.cc:276] 0808 07:02:17.927774 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:17.927814 (+ 40us) service_pool.cc:227] Handling call
0808 07:02:19.469122 (+1541308us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:18.171522 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:18.171540 (+ 18us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:18.171541 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:18.171541 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:18.171552 (+ 11us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:18.171555 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:18.171556 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:18.171582 (+ 26us) tablet.cc:699] Row locks acquired
0808 07:02:18.171583 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:02:18.171590 (+ 7us) write_op.cc:270] Start()
0808 07:02:18.653702 (+482112us) write_op.cc:276] Timestamp: P: 1786172538171540 usec, L: 0
0808 07:02:18.653704 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:02:18.653757 (+ 53us) log.cc:842] Serialized 354 byte log entry
0808 07:02:19.468852 (+815095us) op_driver.cc:444] REPLICATION: finished
0808 07:02:19.469020 (+ 168us) write_op.cc:301] APPLY: starting
0808 07:02:19.469025 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:19.469028 (+ 3us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:19.469032 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:19.469048 (+ 16us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:19.469053 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:19.469053 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:02:19.469076 (+ 23us) log.cc:842] Serialized 22 byte log entry
0808 07:02:19.469095 (+ 19us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:19.469098 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:19.469098 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":152,"num_ops":1,"prepare.queue_time_us":243605,"prepare.run_cpu_time_us":191,"prepare.run_wall_time_us":482256,"replication_time_us":815143,"spinlock_wait_cycles":1280,"thread_start_us":79,"threads_started":1}]]}
I20260808 07:02:19.708609 19483 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:20.300433 19515 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:20.301247 19536 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34320 (ReqId={client: 013957c0297c4f338ebd07fa92c231c7, seq_no=32, attempt_no=0}) took 1647 ms. Trace:
I20260808 07:02:20.301322 19536 rpcz_store.cc:276] 0808 07:02:18.654095 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:18.654117 (+ 22us) service_pool.cc:227] Handling call
0808 07:02:20.301231 (+1647114us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:18.654240 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:18.654253 (+ 13us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:18.654253 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:18.654254 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:18.654264 (+ 10us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:18.654266 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:18.654267 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:18.654275 (+ 8us) tablet.cc:699] Row locks acquired
0808 07:02:18.654276 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:02:18.654281 (+ 5us) write_op.cc:270] Start()
0808 07:02:19.258381 (+604100us) write_op.cc:276] Timestamp: P: 1786172538654266 usec, L: 0
0808 07:02:19.258383 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:02:19.258446 (+ 63us) log.cc:842] Serialized 354 byte log entry
0808 07:02:20.301008 (+1042562us) op_driver.cc:444] REPLICATION: finished
0808 07:02:20.301090 (+ 82us) write_op.cc:301] APPLY: starting
0808 07:02:20.301094 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:20.301099 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:20.301127 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:20.301140 (+ 13us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:20.301145 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:20.301146 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:20.301165 (+ 19us) log.cc:842] Serialized 22 byte log entry
0808 07:02:20.301205 (+ 40us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:20.301208 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:20.301209 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":71,"num_ops":1,"prepare.queue_time_us":38,"prepare.run_cpu_time_us":191,"prepare.run_wall_time_us":604259,"replication_time_us":1042620,"thread_start_us":51,"threads_started":1}]]}
I20260808 07:02:20.393131 19483 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:20.393250 19483 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:20.393510 19483 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:02:20.393692 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:20.393710 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:02:20.393947 18902 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
W20260808 07:02:20.599764 18931 tablet_service.cc:2970] error setting up scanner b66b670c1337474e9b97639de09f2bde with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.206s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "fb12f8b5e19d4b6ca5b7af5f7e1f7574"
I20260808 07:02:20.995509 19483 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:21.003244 19483 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:21.003355 19483 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:21.003517 19483 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:02:21.003656 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:21.003823 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:02:21.004093 18905 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
I20260808 07:02:21.004191 19483 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:21.040514 19546 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34320 (ReqId={client: 013957c0297c4f338ebd07fa92c231c7, seq_no=33, attempt_no=0}) took 1570 ms. Trace:
I20260808 07:02:21.040655 19546 rpcz_store.cc:276] 0808 07:02:19.469922 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:19.469968 (+ 46us) service_pool.cc:227] Handling call
0808 07:02:21.040500 (+1570532us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:19.470122 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:19.470136 (+ 14us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:19.470136 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:19.470136 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:19.470147 (+ 11us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:19.470149 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:19.470150 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:19.470158 (+ 8us) tablet.cc:699] Row locks acquired
0808 07:02:19.470158 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:19.470163 (+ 5us) write_op.cc:270] Start()
0808 07:02:19.866291 (+396128us) write_op.cc:276] Timestamp: P: 1786172539470139 usec, L: 0
0808 07:02:19.866293 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:02:19.866354 (+ 61us) log.cc:842] Serialized 354 byte log entry
0808 07:02:21.040273 (+1173919us) op_driver.cc:444] REPLICATION: finished
0808 07:02:21.040405 (+ 132us) write_op.cc:301] APPLY: starting
0808 07:02:21.040410 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:21.040413 (+ 3us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:21.040416 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:21.040432 (+ 16us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:21.040437 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:21.040437 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:02:21.040458 (+ 21us) log.cc:842] Serialized 22 byte log entry
0808 07:02:21.040476 (+ 18us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:21.040479 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:21.040479 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":118,"num_ops":1,"prepare.queue_time_us":35,"prepare.run_cpu_time_us":184,"prepare.run_wall_time_us":396259,"replication_time_us":1173975,"thread_start_us":72,"threads_started":1}]]}
I20260808 07:02:21.193663 19515 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:21.193789 19515 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:21.193992 19515 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:02:21.194139 18973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:02:21.194125 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:21.194372 19043 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters: 220fdc39470a40e0a6aa355385381500, 6ccea5e821f544dda648f4daccee9bc1
I20260808 07:02:21.988759 19550 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:21.988916 19550 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:21.989568 19550 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:02:21.989737 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:21.989739 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:02:21.989984 18902 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
W20260808 07:02:22.124505 18931 tablet_service.cc:2970] error setting up scanner ea8ac23db1bd4f059945e608c8a828e0 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.135s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "dbf78ad7ad494588bbdc886867884ff8"
I20260808 07:02:22.168776 19515 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:22.701117 19550 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:22.711081 19562 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34320 (ReqId={client: 013957c0297c4f338ebd07fa92c231c7, seq_no=36, attempt_no=0}) took 1644 ms. Trace:
I20260808 07:02:22.711153 19562 rpcz_store.cc:276] 0808 07:02:21.066244 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:21.066280 (+ 36us) service_pool.cc:227] Handling call
0808 07:02:22.711046 (+1644766us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:21.847008 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:21.847026 (+ 18us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:21.847026 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:21.847027 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:21.847042 (+ 15us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:21.847045 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:21.847046 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:21.847057 (+ 11us) tablet.cc:699] Row locks acquired
0808 07:02:21.847057 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:21.847066 (+ 9us) write_op.cc:270] Start()
0808 07:02:21.918242 (+ 71176us) write_op.cc:276] Timestamp: P: 1786172541847019 usec, L: 0
0808 07:02:21.918244 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:02:21.918311 (+ 67us) log.cc:842] Serialized 354 byte log entry
0808 07:02:22.710816 (+792505us) op_driver.cc:444] REPLICATION: finished
0808 07:02:22.710956 (+ 140us) write_op.cc:301] APPLY: starting
0808 07:02:22.710960 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:22.710964 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:22.710967 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:22.710981 (+ 14us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:22.710985 (+ 4us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:22.710985 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:02:22.711005 (+ 20us) log.cc:842] Serialized 22 byte log entry
0808 07:02:22.711021 (+ 16us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:22.711024 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:22.711025 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":126,"num_ops":1,"prepare.queue_time_us":780617,"prepare.run_cpu_time_us":230,"prepare.run_wall_time_us":71345,"replication_time_us":792567,"thread_start_us":84,"threads_started":1}]]}
I20260808 07:02:22.964401 19550 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:22.964547 19550 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:22.964756 19550 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:02:22.964926 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:22.965113 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:02:22.965404 18905 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
I20260808 07:02:22.965596 19550 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:23.052062 19550 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:23.052196 19550 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:23.052436 19550 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:02:23.052579 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:23.052665 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:02:23.052919 18905 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
I20260808 07:02:23.053042 19550 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 07:02:23.416736 18931 tablet_service.cc:2970] error setting up scanner 782e59abd1d94214aca6b2264b1e43a5 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.364s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "fb12f8b5e19d4b6ca5b7af5f7e1f7574"
I20260808 07:02:24.056535 19515 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:24.056663 19515 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:24.056684 19111 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.15.250.131:38731 (request call id 5427) took 1887 ms. Trace:
I20260808 07:02:24.056874 19515 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:02:24.056766 19111 rpcz_store.cc:276] 0808 07:02:22.169305 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:22.169363 (+ 58us) service_pool.cc:227] Handling call
0808 07:02:22.169408 (+ 45us) raft_consensus.cc:1457] Updating replica for 3 ops
0808 07:02:22.169427 (+ 19us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 07:02:22.169428 (+ 1us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0808 07:02:22.169495 (+ 67us) write_op.cc:270] Start()
0808 07:02:22.537621 (+368126us) write_op.cc:276] Timestamp: P: 1786172541041753 usec, L: 0
0808 07:02:22.537749 (+ 128us) write_op.cc:270] Start()
0808 07:02:23.243852 (+706103us) write_op.cc:276] Timestamp: P: 1786172541847019 usec, L: 0
0808 07:02:23.244019 (+ 167us) write_op.cc:270] Start()
0808 07:02:24.056127 (+812108us) write_op.cc:276] Timestamp: P: 1786172542017443 usec, L: 0
0808 07:02:24.056366 (+ 239us) log.cc:842] Serialized 1062 byte log entry
0808 07:02:24.056416 (+ 50us) raft_consensus.cc:1631] Marking committed up to 37
0808 07:02:24.056476 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 07:02:24.056498 (+ 22us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 07:02:24.056653 (+ 155us) raft_consensus.cc:1671] finished
0808 07:02:24.056653 (+ 0us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 07:02:24.056677 (+ 24us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:22.537740 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:22.537757 (+ 17us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:22.537757 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:22.537758 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:22.537780 (+ 22us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:22.537784 (+ 4us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:22.537785 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:22.537794 (+ 9us) tablet.cc:699] Row locks acquired
0808 07:02:22.537794 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:24.056423 (+1518629us) op_driver.cc:444] REPLICATION: finished
0808 07:02:24.056531 (+ 108us) write_op.cc:301] APPLY: starting
0808 07:02:24.056535 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:24.056539 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:24.056539 (+ 0us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:24.056553 (+ 14us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:24.056558 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:24.056561 (+ 3us) write_op.cc:312] APPLY: finished
0808 07:02:24.056576 (+ 15us) log.cc:842] Serialized 22 byte log entry
0808 07:02:24.056581 (+ 5us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:24.056585 (+ 4us) write_op.cc:454] Released schema lock
0808 07:02:24.056585 (+ 0us) write_op.cc:341] FINISH: Updating metrics
0808 07:02:24.056599 (+ 14us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:24.056599 (+ 0us) write_op.cc:454] Released schema lock
Related trace 'op':
0808 07:02:23.244025 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:23.244042 (+ 17us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:23.244043 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:23.244043 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:23.244056 (+ 13us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:23.244059 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:23.244059 (+ 0us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:23.244070 (+ 11us) tablet.cc:699] Row locks acquired
0808 07:02:23.244071 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0808 07:02:24.056453 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:24.056469 (+ 16us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:24.056469 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:24.056470 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:24.056483 (+ 13us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:24.056486 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:24.056487 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:24.056514 (+ 27us) tablet.cc:699] Row locks acquired
0808 07:02:24.056515 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":38,"threads_started":1,"wal-append.queue_time_us":76,"child_traces":[["op",{"apply.queue_time_us":95,"apply.run_cpu_time_us":77,"apply.run_wall_time_us":77,"num_ops":1,"partition_lock_wait_us":2,"prepare.queue_time_us":58,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":78,"replication_time_us":1886925,"thread_start_us":45,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":1,"prepare.queue_time_us":130,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":67,"thread_start_us":102,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":1,"prepare.queue_time_us":271,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":85,"thread_start_us":69,"threads_started":1}]]}
I20260808 07:02:24.057122 18972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:02:24.057147 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:24.057478 19043 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters: 220fdc39470a40e0a6aa355385381500, 6ccea5e821f544dda648f4daccee9bc1
I20260808 07:02:24.057582 19515 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:24.138576 19566 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:24.138700 19566 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:24.138937 19566 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:02:24.139088 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:24.139223 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:02:24.139446 18905 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
I20260808 07:02:24.147517 19578 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34320 (ReqId={client: 013957c0297c4f338ebd07fa92c231c7, seq_no=39, attempt_no=0}) took 1180 ms. Trace:
I20260808 07:02:24.147588 19578 rpcz_store.cc:276] 0808 07:02:22.966661 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:22.966709 (+ 48us) service_pool.cc:227] Handling call
0808 07:02:24.147499 (+1180790us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:23.041284 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:23.041298 (+ 14us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:23.041299 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:23.041299 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:23.041311 (+ 12us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:23.041313 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:23.041314 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:23.041323 (+ 9us) tablet.cc:699] Row locks acquired
0808 07:02:23.041324 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:02:23.041331 (+ 7us) write_op.cc:270] Start()
0808 07:02:23.605464 (+564133us) write_op.cc:276] Timestamp: P: 1786172543041317 usec, L: 0
0808 07:02:23.605465 (+ 1us) op_driver.cc:331] REPLICATION: starting
0808 07:02:23.605514 (+ 49us) log.cc:842] Serialized 354 byte log entry
0808 07:02:24.147216 (+541702us) op_driver.cc:444] REPLICATION: finished
0808 07:02:24.147376 (+ 160us) write_op.cc:301] APPLY: starting
0808 07:02:24.147381 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:24.147385 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:24.147388 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:24.147403 (+ 15us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:24.147407 (+ 4us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:24.147407 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:02:24.147426 (+ 19us) log.cc:842] Serialized 22 byte log entry
0808 07:02:24.147467 (+ 41us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:24.147471 (+ 4us) write_op.cc:454] Released schema lock
0808 07:02:24.147471 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":141,"num_ops":1,"prepare.queue_time_us":74422,"prepare.run_cpu_time_us":154,"prepare.run_wall_time_us":564247,"replication_time_us":541745,"thread_start_us":81,"threads_started":1}]]}
I20260808 07:02:24.451037 19566 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:24.494904 19578 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34320 (ReqId={client: 013957c0297c4f338ebd07fa92c231c7, seq_no=40, attempt_no=0}) took 1419 ms. Trace:
I20260808 07:02:24.494987 19578 rpcz_store.cc:276] 0808 07:02:23.075295 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:23.075316 (+ 21us) service_pool.cc:227] Handling call
0808 07:02:24.494891 (+1419575us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:23.605539 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:23.605552 (+ 13us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:23.605552 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:23.605553 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:23.605585 (+ 32us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:23.605589 (+ 4us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:23.605590 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:23.605600 (+ 10us) tablet.cc:699] Row locks acquired
0808 07:02:23.605601 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:02:23.605609 (+ 8us) write_op.cc:270] Start()
0808 07:02:23.715737 (+110128us) write_op.cc:276] Timestamp: P: 1786172543605581 usec, L: 0
0808 07:02:23.715739 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:02:23.715801 (+ 62us) log.cc:842] Serialized 354 byte log entry
0808 07:02:24.494698 (+778897us) op_driver.cc:444] REPLICATION: finished
0808 07:02:24.494780 (+ 82us) write_op.cc:301] APPLY: starting
0808 07:02:24.494785 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:24.494791 (+ 6us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:24.494795 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:24.494814 (+ 19us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:24.494819 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:24.494820 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:24.494846 (+ 26us) log.cc:842] Serialized 22 byte log entry
0808 07:02:24.494863 (+ 17us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:24.494867 (+ 4us) write_op.cc:454] Released schema lock
0808 07:02:24.494867 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":41,"num_ops":1,"prepare.queue_time_us":530122,"prepare.run_cpu_time_us":199,"prepare.run_wall_time_us":110288,"replication_time_us":778953}]]}
I20260808 07:02:24.898147 19566 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:24.898902 19566 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:24.900166 19566 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:02:24.900408 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:24.900611 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:02:24.901010 18905 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
I20260808 07:02:25.138098 19566 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:26.045926 19515 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:26.046036 19515 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:26.046052 19111 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.15.250.131:38731 (request call id 5428) took 1988 ms. Trace:
I20260808 07:02:26.046317 19515 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:02:26.046175 19111 rpcz_store.cc:276] 0808 07:02:24.057649 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:24.057674 (+ 25us) service_pool.cc:227] Handling call
0808 07:02:24.057717 (+ 43us) raft_consensus.cc:1457] Updating replica for 3 ops
0808 07:02:24.057745 (+ 28us) raft_consensus.cc:1520] Early marking committed up to index 39
0808 07:02:24.057808 (+ 63us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0808 07:02:24.057844 (+ 36us) write_op.cc:270] Start()
0808 07:02:24.407009 (+349165us) write_op.cc:276] Timestamp: P: 1786172542712081 usec, L: 0
0808 07:02:24.407129 (+ 120us) write_op.cc:270] Start()
0808 07:02:25.052275 (+645146us) write_op.cc:276] Timestamp: P: 1786172543041317 usec, L: 0
0808 07:02:25.052461 (+ 186us) write_op.cc:270] Start()
0808 07:02:26.045584 (+993123us) write_op.cc:276] Timestamp: P: 1786172543605581 usec, L: 0
0808 07:02:26.045749 (+ 165us) log.cc:842] Serialized 1062 byte log entry
0808 07:02:26.045829 (+ 80us) raft_consensus.cc:1631] Marking committed up to 40
0808 07:02:26.045878 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 07:02:26.045896 (+ 18us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 07:02:26.046021 (+ 125us) raft_consensus.cc:1671] finished
0808 07:02:26.046022 (+ 1us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 07:02:26.046033 (+ 11us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:24.407120 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:24.407136 (+ 16us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:24.407137 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:24.407137 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:24.407152 (+ 15us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:24.407156 (+ 4us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:24.407160 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:24.407174 (+ 14us) tablet.cc:699] Row locks acquired
0808 07:02:24.407175 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:02:26.045836 (+1638661us) op_driver.cc:444] REPLICATION: finished
0808 07:02:26.045928 (+ 92us) write_op.cc:301] APPLY: starting
0808 07:02:26.045931 (+ 3us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:26.045936 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:26.045937 (+ 1us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:26.045952 (+ 15us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:26.045957 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:26.045960 (+ 3us) write_op.cc:312] APPLY: finished
0808 07:02:26.045975 (+ 15us) log.cc:842] Serialized 22 byte log entry
0808 07:02:26.045981 (+ 6us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:26.045983 (+ 2us) write_op.cc:454] Released schema lock
0808 07:02:26.045984 (+ 1us) write_op.cc:341] FINISH: Updating metrics
0808 07:02:26.045998 (+ 14us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:26.045998 (+ 0us) write_op.cc:454] Released schema lock
Related trace 'op':
0808 07:02:25.052481 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:25.052498 (+ 17us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:25.052498 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:25.052499 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:25.052512 (+ 13us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:25.052514 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:25.052515 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:25.052526 (+ 11us) tablet.cc:699] Row locks acquired
0808 07:02:25.052526 (+ 0us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0808 07:02:26.045806 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:26.045823 (+ 17us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:26.045823 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:26.045824 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:26.045836 (+ 12us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:26.045839 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:26.045840 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:26.045849 (+ 9us) tablet.cc:699] Row locks acquired
0808 07:02:26.045850 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":70,"threads_started":1,"wal-append.queue_time_us":96,"child_traces":[["op",{"apply.queue_time_us":84,"apply.run_cpu_time_us":80,"apply.run_wall_time_us":81,"num_ops":1,"partition_lock_wait_us":2,"prepare.queue_time_us":57,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":80,"replication_time_us":1987989,"thread_start_us":35,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":1,"prepare.queue_time_us":139,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":74,"thread_start_us":99,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":0,"prepare.queue_time_us":171,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":61,"thread_start_us":72,"threads_started":1}]]}
I20260808 07:02:26.046525 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:26.046546 18972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:02:26.046777 19043 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters: 220fdc39470a40e0a6aa355385381500, 6ccea5e821f544dda648f4daccee9bc1
I20260808 07:02:26.046897 19515 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:26.217042 19580 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:26.217185 19580 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:26.217494 19580 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:02:26.217681 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:26.217723 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:02:26.218014 18902 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
I20260808 07:02:26.218137 19580 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:26.237069 19590 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34320 (ReqId={client: 013957c0297c4f338ebd07fa92c231c7, seq_no=43, attempt_no=0}) took 1324 ms. Trace:
I20260808 07:02:26.237164 19590 rpcz_store.cc:276] 0808 07:02:24.912813 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:24.912850 (+ 37us) service_pool.cc:227] Handling call
0808 07:02:26.237055 (+1324205us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:24.913016 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:24.913029 (+ 13us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:24.913029 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:24.913030 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:24.913049 (+ 19us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:24.913051 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:24.913052 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:24.913061 (+ 9us) tablet.cc:699] Row locks acquired
0808 07:02:24.913062 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:02:24.913066 (+ 4us) write_op.cc:270] Start()
0808 07:02:25.726175 (+813109us) write_op.cc:276] Timestamp: P: 1786172544913028 usec, L: 0
0808 07:02:25.726176 (+ 1us) op_driver.cc:331] REPLICATION: starting
0808 07:02:25.726224 (+ 48us) log.cc:842] Serialized 354 byte log entry
0808 07:02:26.236702 (+510478us) op_driver.cc:444] REPLICATION: finished
0808 07:02:26.236828 (+ 126us) write_op.cc:301] APPLY: starting
0808 07:02:26.236834 (+ 6us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:26.236839 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:26.236842 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:26.236858 (+ 16us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:26.236949 (+ 91us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:26.236950 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:26.237001 (+ 51us) log.cc:842] Serialized 22 byte log entry
0808 07:02:26.237024 (+ 23us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:26.237029 (+ 5us) write_op.cc:454] Released schema lock
0808 07:02:26.237030 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":112,"num_ops":1,"prepare.queue_time_us":34,"prepare.run_cpu_time_us":171,"prepare.run_wall_time_us":813246,"replication_time_us":510517,"thread_start_us":82,"threads_started":1}]]}
I20260808 07:02:26.681180 19580 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:26.681317 19580 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:26.681553 19580 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:02:26.681735 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:26.681773 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:02:26.682076 18905 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
I20260808 07:02:26.682179 19580 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:26.692996 19590 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34320 (ReqId={client: 013957c0297c4f338ebd07fa92c231c7, seq_no=44, attempt_no=0}) took 1539 ms. Trace:
I20260808 07:02:26.693208 19590 rpcz_store.cc:276] 0808 07:02:25.153303 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:25.153323 (+ 20us) service_pool.cc:227] Handling call
0808 07:02:26.692985 (+1539662us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:25.726272 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:25.726289 (+ 17us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:25.726290 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:25.726291 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:25.726303 (+ 12us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:25.726306 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:25.726307 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:25.726318 (+ 11us) tablet.cc:699] Row locks acquired
0808 07:02:25.726318 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:25.726327 (+ 9us) write_op.cc:270] Start()
0808 07:02:26.244421 (+518094us) write_op.cc:276] Timestamp: P: 1786172545726295 usec, L: 0
0808 07:02:26.244422 (+ 1us) op_driver.cc:331] REPLICATION: starting
0808 07:02:26.244468 (+ 46us) log.cc:842] Serialized 354 byte log entry
0808 07:02:26.692767 (+448299us) op_driver.cc:444] REPLICATION: finished
0808 07:02:26.692847 (+ 80us) write_op.cc:301] APPLY: starting
0808 07:02:26.692851 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:26.692857 (+ 6us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:26.692863 (+ 6us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:26.692886 (+ 23us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:26.692892 (+ 6us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:26.692892 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:02:26.692916 (+ 24us) log.cc:842] Serialized 22 byte log entry
0808 07:02:26.692961 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:26.692963 (+ 2us) write_op.cc:454] Released schema lock
0808 07:02:26.692964 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":53,"mutex_wait_us":56,"num_ops":1,"prepare.queue_time_us":572862,"prepare.run_cpu_time_us":194,"prepare.run_wall_time_us":518304,"raft.queue_time_us":77,"raft.run_cpu_time_us":2,"raft.run_wall_time_us":2,"replication_time_us":448340}]]}
I20260808 07:02:27.104902 19580 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:27.105027 19580 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:27.105245 19580 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:02:27.105448 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:27.105518 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:02:27.105752 18905 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
W20260808 07:02:27.106897 19342 scanner-internal.cc:458] Time spent opening tablet: real 29.051s user 0.003s sys 0.000s
W20260808 07:02:27.257768 18931 tablet_service.cc:2970] error setting up scanner fbb32fb0b24648cb97618626b4a65bb7 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.152s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7316162752948211712 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b5ff112fd6bc48269c95da159bf34c2c"
W20260808 07:02:27.271095 18931 tablet_service.cc:2970] error setting up scanner 37d2baf4269f425586230bbc290561fc with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.165s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7316162752948211712 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b5ff112fd6bc48269c95da159bf34c2c"
W20260808 07:02:27.300233 18931 tablet_service.cc:2970] error setting up scanner d271d3f730874a62b73236526f5494ea with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.194s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7316162752948211712 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b5ff112fd6bc48269c95da159bf34c2c"
W20260808 07:02:27.365550 18931 tablet_service.cc:2970] error setting up scanner 7a029e2400354c8cbdc25cc0592d47e3 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.260s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7316162752948211712 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b5ff112fd6bc48269c95da159bf34c2c"
W20260808 07:02:27.502739 18931 tablet_service.cc:2970] error setting up scanner 65f24b52255b4ae9990cef5d5b3cd7f7 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.397s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7316162752948211712 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b5ff112fd6bc48269c95da159bf34c2c"
W20260808 07:02:27.513792 18931 tablet_service.cc:2970] error setting up scanner 4221610e7ad0445ea117238db0f067b0 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.408s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "fb12f8b5e19d4b6ca5b7af5f7e1f7574"
W20260808 07:02:27.663805 18931 tablet_service.cc:2970] error setting up scanner fa53ea994a25412a9296105cbc8e285b with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.558s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7316162752948211712 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b5ff112fd6bc48269c95da159bf34c2c"
I20260808 07:02:27.842278 19580 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:27.877342 19597 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34320 (ReqId={client: 013957c0297c4f338ebd07fa92c231c7, seq_no=46, attempt_no=0}) took 1183 ms. Trace:
I20260808 07:02:27.877470 19597 rpcz_store.cc:276] 0808 07:02:26.694010 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:26.694050 (+ 40us) service_pool.cc:227] Handling call
0808 07:02:27.877317 (+1183267us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:26.899854 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:26.899894 (+ 40us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:26.899895 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:26.899895 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:26.899908 (+ 13us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:26.899911 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:26.899912 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:26.899923 (+ 11us) tablet.cc:699] Row locks acquired
0808 07:02:26.899923 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:26.899931 (+ 8us) write_op.cc:270] Start()
0808 07:02:27.015037 (+115106us) write_op.cc:276] Timestamp: P: 1786172546899891 usec, L: 0
0808 07:02:27.015040 (+ 3us) op_driver.cc:331] REPLICATION: starting
0808 07:02:27.015105 (+ 65us) log.cc:842] Serialized 354 byte log entry
0808 07:02:27.876934 (+861829us) op_driver.cc:444] REPLICATION: finished
0808 07:02:27.877099 (+ 165us) write_op.cc:301] APPLY: starting
0808 07:02:27.877104 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:27.877108 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:27.877111 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:27.877128 (+ 17us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:27.877133 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:27.877133 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:02:27.877157 (+ 24us) log.cc:842] Serialized 22 byte log entry
0808 07:02:27.877269 (+ 112us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:27.877272 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:27.877272 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":149,"num_ops":1,"prepare.queue_time_us":205654,"prepare.run_cpu_time_us":204,"prepare.run_wall_time_us":115275,"replication_time_us":861888,"thread_start_us":81,"threads_started":1}]]}
I20260808 07:02:27.948791 19515 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:27.948887 19515 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:27.949101 19515 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:02:27.949235 19111 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.15.250.131:38731 (request call id 5429) took 1902 ms. Trace:
I20260808 07:02:27.949347 18972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:02:27.949349 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:27.949285 19111 rpcz_store.cc:276] 0808 07:02:26.047016 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:26.047038 (+ 22us) service_pool.cc:227] Handling call
0808 07:02:26.047092 (+ 54us) raft_consensus.cc:1457] Updating replica for 3 ops
0808 07:02:26.047106 (+ 14us) raft_consensus.cc:1520] Early marking committed up to index 42
0808 07:02:26.047204 (+ 98us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0808 07:02:26.047246 (+ 42us) write_op.cc:270] Start()
0808 07:02:26.087346 (+ 40100us) write_op.cc:276] Timestamp: P: 1786172544148421 usec, L: 0
0808 07:02:26.087425 (+ 79us) write_op.cc:270] Start()
0808 07:02:27.038577 (+951152us) write_op.cc:276] Timestamp: P: 1786172544495977 usec, L: 0
0808 07:02:27.038784 (+ 207us) write_op.cc:270] Start()
0808 07:02:27.948381 (+909597us) write_op.cc:276] Timestamp: P: 1786172544913028 usec, L: 0
0808 07:02:27.948585 (+ 204us) log.cc:842] Serialized 1062 byte log entry
0808 07:02:27.948647 (+ 62us) raft_consensus.cc:1631] Marking committed up to 44
0808 07:02:27.948755 (+ 108us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 07:02:27.948767 (+ 12us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 07:02:27.949215 (+ 448us) raft_consensus.cc:1671] finished
0808 07:02:27.949216 (+ 1us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 07:02:27.949229 (+ 13us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:26.087456 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:26.087470 (+ 14us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:26.087471 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:26.087471 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:26.087487 (+ 16us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:26.087491 (+ 4us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:26.087492 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:26.087509 (+ 17us) tablet.cc:699] Row locks acquired
0808 07:02:26.087509 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:27.948655 (+1861146us) op_driver.cc:444] REPLICATION: finished
0808 07:02:27.948738 (+ 83us) write_op.cc:301] APPLY: starting
0808 07:02:27.948742 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:27.948747 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:27.948747 (+ 0us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:27.948761 (+ 14us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:27.948767 (+ 6us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:27.948770 (+ 3us) write_op.cc:312] APPLY: finished
0808 07:02:27.948786 (+ 16us) log.cc:842] Serialized 22 byte log entry
0808 07:02:27.948791 (+ 5us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:27.948794 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:27.948794 (+ 0us) write_op.cc:341] FINISH: Updating metrics
0808 07:02:27.948810 (+ 16us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:27.948811 (+ 1us) write_op.cc:454] Released schema lock
Related trace 'op':
0808 07:02:27.038841 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:27.038858 (+ 17us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:27.038858 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:27.038859 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:27.038872 (+ 13us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:27.038875 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:27.038876 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:27.038886 (+ 10us) tablet.cc:699] Row locks acquired
0808 07:02:27.038886 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:27.948720 (+909834us) op_driver.cc:444] REPLICATION: finished
0808 07:02:27.948825 (+ 105us) write_op.cc:301] APPLY: starting
0808 07:02:27.948826 (+ 1us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:27.948828 (+ 2us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:27.948828 (+ 0us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:27.948832 (+ 4us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:27.948833 (+ 1us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:27.948835 (+ 2us) write_op.cc:312] APPLY: finished
0808 07:02:27.948838 (+ 3us) log.cc:842] Serialized 22 byte log entry
0808 07:02:27.948840 (+ 2us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:27.948841 (+ 1us) write_op.cc:454] Released schema lock
0808 07:02:27.948841 (+ 0us) write_op.cc:341] FINISH: Updating metrics
0808 07:02:27.948846 (+ 5us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:27.948846 (+ 0us) write_op.cc:454] Released schema lock
Related trace 'op':
0808 07:02:27.948561 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:27.948578 (+ 17us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:27.948579 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:27.948579 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:27.948602 (+ 23us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:27.948605 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:27.948607 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:27.948623 (+ 16us) tablet.cc:699] Row locks acquired
0808 07:02:27.948623 (+ 0us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":46,"threads_started":1,"wal-append.queue_time_us":88,"child_traces":[["op",{"apply.queue_time_us":73,"apply.run_cpu_time_us":85,"apply.run_wall_time_us":84,"num_ops":1,"partition_lock_wait_us":2,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":92,"replication_time_us":1901407,"thread_start_us":56,"threads_started":1}],["op",{"apply.queue_time_us":102,"apply.run_cpu_time_us":26,"apply.run_wall_time_us":26,"num_ops":1,"partition_lock_wait_us":0,"prepare.queue_time_us":206,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":73,"replication_time_us":1861293,"thread_start_us":160,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":1,"prepare.queue_time_us":132,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":83,"thread_start_us":97,"threads_started":1}]]}
I20260808 07:02:27.949654 19043 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters: 220fdc39470a40e0a6aa355385381500, 6ccea5e821f544dda648f4daccee9bc1
I20260808 07:02:27.949960 19515 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 07:02:28.080829 18931 tablet_service.cc:2970] error setting up scanner ad10b6291fd44949a643473e8d84db89 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.238s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7316162752948211712 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b5ff112fd6bc48269c95da159bf34c2c"
I20260808 07:02:28.475618 19580 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:28.475725 19580 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:28.475981 19580 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:02:28.476158 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 49 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:28.476186 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 49 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:02:28.476488 18902 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
I20260808 07:02:28.487768 19606 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34320 (ReqId={client: 013957c0297c4f338ebd07fa92c231c7, seq_no=47, attempt_no=0}) took 1378 ms. Trace:
I20260808 07:02:28.487855 19606 rpcz_store.cc:276] 0808 07:02:27.109244 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:27.109331 (+ 87us) service_pool.cc:227] Handling call
0808 07:02:28.487743 (+1378412us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:27.109512 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:27.109523 (+ 11us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:27.109524 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:27.109524 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:27.109537 (+ 13us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:27.109538 (+ 1us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:27.109539 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:27.109547 (+ 8us) tablet.cc:699] Row locks acquired
0808 07:02:27.109547 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:27.109553 (+ 6us) write_op.cc:270] Start()
0808 07:02:27.388669 (+279116us) write_op.cc:276] Timestamp: P: 1786172547109496 usec, L: 0
0808 07:02:27.388671 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:02:27.388746 (+ 75us) log.cc:842] Serialized 354 byte log entry
0808 07:02:28.487319 (+1098573us) op_driver.cc:444] REPLICATION: finished
0808 07:02:28.487493 (+ 174us) write_op.cc:301] APPLY: starting
0808 07:02:28.487499 (+ 6us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:28.487504 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:28.487518 (+ 14us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:28.487537 (+ 19us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:28.487543 (+ 6us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:28.487543 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:02:28.487567 (+ 24us) log.cc:842] Serialized 22 byte log entry
0808 07:02:28.487705 (+ 138us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:28.487713 (+ 8us) write_op.cc:454] Released schema lock
0808 07:02:28.487714 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":158,"num_ops":1,"prepare.queue_time_us":39,"prepare.run_cpu_time_us":194,"prepare.run_wall_time_us":279270,"replication_time_us":1098642,"thread_start_us":104,"threads_started":1}]]}
W20260808 07:02:28.785902 18931 tablet_service.cc:2970] error setting up scanner ec99c5e86d714c15b414a8271ad923c0 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.309s (maximum allowed 0.100s): new_scan_request { tablet_id: "3867cda74ea345ef86c68d815680f9dd" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7316162752948211712 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\221\253\333\323\006\"3\n\005slave\022*\n 3297866f91a041eaa0f9823439545d32\020\001\030\001 \001(\001" signature: "<redacted>""e>\346\002\225\250T\377\306-\255\275\227\253r\356\331\210\205\212.\370\357\237\270\016z\234j^xX\226\334\366n\005\333\330\2157\002a{z\352\267<\316\264\324\n\034\0208\344j\300\366ThT\217\322" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b5ff112fd6bc48269c95da159bf34c2c"
I20260808 07:02:28.980006 19111 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.15.250.131:38731 (request call id 5430) took 1029 ms. Trace:
I20260808 07:02:28.980013 19515 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:28.980239 19515 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:28.980108 19111 rpcz_store.cc:276] 0808 07:02:27.950548 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:27.950606 (+ 58us) service_pool.cc:227] Handling call
0808 07:02:27.950683 (+ 77us) raft_consensus.cc:1457] Updating replica for 4 ops
0808 07:02:27.950704 (+ 21us) raft_consensus.cc:1520] Early marking committed up to index 45
0808 07:02:27.950725 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 4 ops
0808 07:02:27.950776 (+ 51us) write_op.cc:270] Start()
0808 07:02:27.952866 (+ 2090us) write_op.cc:276] Timestamp: P: 1786172545726295 usec, L: 0
0808 07:02:27.952939 (+ 73us) write_op.cc:270] Start()
0808 07:02:28.790049 (+837110us) write_op.cc:276] Timestamp: P: 1786172546244609 usec, L: 0
0808 07:02:28.790215 (+ 166us) write_op.cc:270] Start()
0808 07:02:28.968353 (+178138us) write_op.cc:276] Timestamp: P: 1786172546899891 usec, L: 0
0808 07:02:28.968454 (+ 101us) write_op.cc:270] Start()
0808 07:02:28.979551 (+ 11097us) write_op.cc:276] Timestamp: P: 1786172547109496 usec, L: 0
0808 07:02:28.979655 (+ 104us) log.cc:842] Serialized 1416 byte log entry
0808 07:02:28.979755 (+ 100us) raft_consensus.cc:1631] Marking committed up to 48
0808 07:02:28.979966 (+ 211us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 07:02:28.979976 (+ 10us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 07:02:28.979978 (+ 2us) raft_consensus.cc:1671] finished
0808 07:02:28.979979 (+ 1us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 07:02:28.979992 (+ 13us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:27.952941 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:27.952955 (+ 14us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:27.952955 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:27.952955 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:27.952969 (+ 14us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:27.952976 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:27.952977 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:27.952984 (+ 7us) tablet.cc:699] Row locks acquired
0808 07:02:27.952985 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:02:28.979763 (+1026778us) op_driver.cc:444] REPLICATION: finished
0808 07:02:28.979873 (+ 110us) write_op.cc:301] APPLY: starting
0808 07:02:28.979877 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:28.979882 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:28.979882 (+ 0us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:28.979897 (+ 15us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:28.979903 (+ 6us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:28.979907 (+ 4us) write_op.cc:312] APPLY: finished
0808 07:02:28.979923 (+ 16us) log.cc:842] Serialized 22 byte log entry
0808 07:02:28.979929 (+ 6us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:28.979932 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:28.979932 (+ 0us) write_op.cc:341] FINISH: Updating metrics
0808 07:02:28.979950 (+ 18us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:28.979951 (+ 1us) write_op.cc:454] Released schema lock
Related trace 'op':
0808 07:02:28.790238 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:28.790256 (+ 18us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:28.790257 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:28.790257 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:28.790269 (+ 12us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:28.790272 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:28.790273 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:28.790283 (+ 10us) tablet.cc:699] Row locks acquired
0808 07:02:28.790283 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:28.979836 (+189553us) op_driver.cc:444] REPLICATION: finished
0808 07:02:28.979967 (+ 131us) write_op.cc:301] APPLY: starting
0808 07:02:28.979968 (+ 1us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:28.979969 (+ 1us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:28.979969 (+ 0us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:28.979973 (+ 4us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:28.979975 (+ 2us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:28.979977 (+ 2us) write_op.cc:312] APPLY: finished
0808 07:02:28.979981 (+ 4us) log.cc:842] Serialized 22 byte log entry
0808 07:02:28.980012 (+ 31us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:28.980013 (+ 1us) write_op.cc:454] Released schema lock
0808 07:02:28.980013 (+ 0us) write_op.cc:341] FINISH: Updating metrics
0808 07:02:28.980018 (+ 5us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:28.980019 (+ 1us) write_op.cc:454] Released schema lock
Related trace 'op':
0808 07:02:28.968477 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:28.968497 (+ 20us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:28.968498 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:28.968498 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:28.968525 (+ 27us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:28.968530 (+ 5us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:28.968531 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:28.968543 (+ 12us) tablet.cc:699] Row locks acquired
0808 07:02:28.968543 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:28.979873 (+ 11330us) op_driver.cc:444] REPLICATION: finished
0808 07:02:28.980030 (+ 157us) write_op.cc:301] APPLY: starting
0808 07:02:28.980030 (+ 0us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:28.980032 (+ 2us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:28.980032 (+ 0us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:28.980034 (+ 2us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:28.980036 (+ 2us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:28.980037 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:28.980042 (+ 5us) log.cc:842] Serialized 22 byte log entry
0808 07:02:28.980044 (+ 2us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:28.980045 (+ 1us) write_op.cc:454] Released schema lock
0808 07:02:28.980045 (+ 0us) write_op.cc:341] FINISH: Updating metrics
0808 07:02:28.980050 (+ 5us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:28.980050 (+ 0us) write_op.cc:454] Released schema lock
Related trace 'op':
0808 07:02:28.979657 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:28.979680 (+ 23us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:28.979680 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:28.979681 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:28.979694 (+ 13us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:28.979697 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:28.979698 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:28.979709 (+ 11us) tablet.cc:699] Row locks acquired
0808 07:02:28.979709 (+ 0us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":72,"threads_started":1,"wal-append.queue_time_us":117,"child_traces":[["op",{"apply.queue_time_us":98,"apply.run_cpu_time_us":88,"apply.run_wall_time_us":90,"num_ops":1,"partition_lock_wait_us":1,"prepare.queue_time_us":43,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":60,"replication_time_us":1028984,"thread_start_us":59,"threads_started":1}],["op",{"apply.queue_time_us":127,"apply.run_cpu_time_us":59,"apply.run_wall_time_us":59,"num_ops":1,"partition_lock_wait_us":1,"prepare.queue_time_us":135,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":67,"replication_time_us":1026896,"thread_start_us":124,"threads_started":2}],["op",{"apply.queue_time_us":153,"apply.run_cpu_time_us":26,"apply.run_wall_time_us":26,"mutex_wait_us":29,"num_ops":1,"partition_lock_wait_us":1,"prepare.queue_time_us":58,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":86,"replication_time_us":189656,"thread_start_us":89,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":1,"prepare.queue_time_us":52,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":70}]]}
I20260808 07:02:28.980527 19515 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:02:28.980773 18972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 49 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:02:28.980780 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 49 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:28.981019 19043 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters: 220fdc39470a40e0a6aa355385381500, 6ccea5e821f544dda648f4daccee9bc1
I20260808 07:02:28.981119 19515 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:28.985917 19580 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:28.987676 19606 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34320 (ReqId={client: 013957c0297c4f338ebd07fa92c231c7, seq_no=48, attempt_no=0}) took 1109 ms. Trace:
I20260808 07:02:28.987753 19606 rpcz_store.cc:276] 0808 07:02:27.878158 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:27.878197 (+ 39us) service_pool.cc:227] Handling call
0808 07:02:28.987666 (+1109469us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:27.878318 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3867cda74ea345ef86c68d815680f9dd
0808 07:02:27.878330 (+ 12us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:27.878331 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:27.878331 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:27.878341 (+ 10us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:27.878343 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:27.878345 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:27.878353 (+ 8us) tablet.cc:699] Row locks acquired
0808 07:02:27.878353 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:27.878360 (+ 7us) write_op.cc:270] Start()
0808 07:02:28.468484 (+590124us) write_op.cc:276] Timestamp: P: 1786172547878323 usec, L: 0
0808 07:02:28.468486 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:02:28.468562 (+ 76us) log.cc:842] Serialized 354 byte log entry
0808 07:02:28.987486 (+518924us) op_driver.cc:444] REPLICATION: finished
0808 07:02:28.987555 (+ 69us) write_op.cc:301] APPLY: starting
0808 07:02:28.987558 (+ 3us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:28.987561 (+ 3us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:28.987563 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:28.987574 (+ 11us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:28.987578 (+ 4us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:28.987578 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:02:28.987594 (+ 16us) log.cc:842] Serialized 22 byte log entry
0808 07:02:28.987640 (+ 46us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:28.987643 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:28.987644 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":52,"num_ops":1,"prepare.queue_time_us":33,"prepare.run_cpu_time_us":218,"prepare.run_wall_time_us":590274,"replication_time_us":518996}]]}
I20260808 07:02:29.209620 19515 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:29.209746 19515 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:29.209949 19515 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:02:29.210146 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:29.210279 18973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:02:29.210602 19041 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters: 220fdc39470a40e0a6aa355385381500, 6ccea5e821f544dda648f4daccee9bc1
I20260808 07:02:29.210748 19515 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 07:02:30.075886 19341 scanner-internal.cc:458] Time spent opening tablet: real 32.020s user 0.003s sys 0.000s
W20260808 07:02:30.452661 19342 scanner-internal.cc:458] Time spent opening tablet: real 3.195s user 0.002s sys 0.000s
I20260808 07:02:30.468719 19623 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:30.468528 19622 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:30.468793 19622 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:30.468793 19623 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:30.469076 19622 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:02:30.469112 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:30.469164 18973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:02:30.469189 19250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:30.469646 19041 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters: 220fdc39470a40e0a6aa355385381500, 6ccea5e821f544dda648f4daccee9bc1
I20260808 07:02:30.469794 19622 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:30.469918 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6ccea5e821f544dda648f4daccee9bc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" is_pre_election: true
I20260808 07:02:30.470479 18905 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ccea5e821f544dda648f4daccee9bc1; no voters: 220fdc39470a40e0a6aa355385381500, 6fbba1bb0f04471ba1cfa1a42b8619cf
I20260808 07:02:30.470872 19623 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995)
I20260808 07:02:30.470907 19624 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6ccea5e821f544dda648f4daccee9bc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:30.504403 18951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260808 07:02:30.504599 19229 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260808 07:02:30.509665 19089 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260808 07:02:30.512151 19622 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:30.512224 19622 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:30.512563 19622 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:02:30.512743 19250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
I20260808 07:02:30.514891 18973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "6ccea5e821f544dda648f4daccee9bc1" is_pre_election: true
I20260808 07:02:30.515161 19043 leader_election.cc:302] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fbba1bb0f04471ba1cfa1a42b8619cf; no voters: 220fdc39470a40e0a6aa355385381500, 6ccea5e821f544dda648f4daccee9bc1
I20260808 07:02:30.515311 19622 raft_consensus.cc:2726] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
99990c52528042d783bfc0b3815c0715 | 127.15.250.190:35249 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.250.148:33287 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
220fdc39470a40e0a6aa355385381500 | 127.15.250.131:44619 | HEALTHY | <none> | 1 | 0
6ccea5e821f544dda648f4daccee9bc1 | 127.15.250.129:46595 | HEALTHY | <none> | 0 | 0
6fbba1bb0f04471ba1cfa1a42b8619cf | 127.15.250.130:36995 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
consensus_inject_latency_ms_in_notifications | 50 | unsafe,hidden | all 3 server(s) checked
inject_latency_ms_before_starting_op | 1000 | hidden,advanced | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.15.250.129 | experimental | 127.15.250.129:46595
local_ip_for_outbound_sockets | 127.15.250.130 | experimental | 127.15.250.130:36995
local_ip_for_outbound_sockets | 127.15.250.131 | experimental | 127.15.250.131:44619
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.15.250.129:46595
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.15.250.130:36995
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.15.250.131:44619
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.250.148:33287 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260808 07:02:30.522070 16362 log_verifier.cc:126] Checking tablet 3867cda74ea345ef86c68d815680f9dd
I20260808 07:02:30.545672 16362 log_verifier.cc:177] Verified matching terms for 50 ops in tablet 3867cda74ea345ef86c68d815680f9dd
I20260808 07:02:30.558382 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 18886
I20260808 07:02:30.566705 19622 raft_consensus.cc:482] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 220fdc39470a40e0a6aa355385381500)
I20260808 07:02:30.566814 19622 raft_consensus.cc:504] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220fdc39470a40e0a6aa355385381500" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44619 } } peers { permanent_uuid: "6ccea5e821f544dda648f4daccee9bc1" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 46595 } } peers { permanent_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36995 } }
I20260808 07:02:30.566989 19622 leader_election.cc:288] T 3867cda74ea345ef86c68d815680f9dd P 6fbba1bb0f04471ba1cfa1a42b8619cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220fdc39470a40e0a6aa355385381500 (127.15.250.131:44619), 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595)
I20260808 07:02:30.568449 19250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3867cda74ea345ef86c68d815680f9dd" candidate_uuid: "6fbba1bb0f04471ba1cfa1a42b8619cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "220fdc39470a40e0a6aa355385381500" is_pre_election: true
W20260808 07:02:30.572299 19183 connection.cc:535] client connection to 127.15.250.129:46595 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 07:02:30.572399 19043 connection.cc:535] client connection to 127.15.250.129:46595 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 07:02:30.572449 19043 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 1 similar messages]
W20260808 07:02:30.572438 19183 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 07:02:30.572839 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 19025
W20260808 07:02:30.574981 19183 consensus_peers.cc:487] T 3867cda74ea345ef86c68d815680f9dd P 220fdc39470a40e0a6aa355385381500 -> Peer 6ccea5e821f544dda648f4daccee9bc1 (127.15.250.129:46595): Couldn't send request to peer 6ccea5e821f544dda648f4daccee9bc1. Status: Network error: Client connection negotiation failed: client connection to 127.15.250.129:46595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260808 07:02:30.579296 19184 connection.cc:535] client connection to 127.15.250.130:36995 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 07:02:30.579725 19184 consensus_peers.cc:487] T 3867cda74ea345ef86c68d815680f9dd P 220fdc39470a40e0a6aa355385381500 -> Peer 6fbba1bb0f04471ba1cfa1a42b8619cf (127.15.250.130:36995): Couldn't send request to peer 6fbba1bb0f04471ba1cfa1a42b8619cf. Status: Network error: Client connection negotiation failed: client connection to 127.15.250.130:36995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 07:02:30.579821 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 19164
I20260808 07:02:30.586863 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 18787
2026-08-08T07:02:30Z chronyd exiting
[ OK ] RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency (33326 ms)
[ RUN ] RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys
I20260808 07:02:30.611452 16362 test_util.cc:268] Using random seed: -1268579850
I20260808 07:02:30.612010 16362 ts_itest-base.cc:112] Starting cluster with:
I20260808 07:02:30.612056 16362 ts_itest-base.cc:113] --------------
I20260808 07:02:30.612068 16362 ts_itest-base.cc:114] 3 tablet servers
I20260808 07:02:30.612078 16362 ts_itest-base.cc:115] 3 replicas per TS
I20260808 07:02:30.612089 16362 ts_itest-base.cc:116] --------------
2026-08-08T07:02:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:02:30Z Disabled control of system clock
I20260808 07:02:30.622253 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.250.190:43327
--webserver_interface=127.15.250.190
--webserver_port=0
--builtin_ntp_servers=127.15.250.148:39425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.250.190:43327 with env {}
W20260808 07:02:30.706393 19665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:02:30.706635 19665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:02:30.706660 19665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:02:30.707813 19665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:02:30.707851 19665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:02:30.707865 19665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:02:30.707877 19665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:02:30.709270 19665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:39425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.250.190:43327
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.250.190:43327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.250.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.19665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:02:30.709496 19665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:02:30.709795 19665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:02:30.715039 19671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:02:30.715060 19673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:02:30.715045 19675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:02:30.715445 19665 server_base.cc:1034] running on GCE node
I20260808 07:02:30.715723 19665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:02:30.716050 19665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:02:30.717226 19665 hybrid_clock.cc:648] HybridClock initialized: now 1786172550717201 us; error 59 us; skew 500 ppm
I20260808 07:02:30.718678 19665 webserver.cc:460] Webserver started at http://127.15.250.190:35553/ using document root <none> and password file <none>
I20260808 07:02:30.718966 19665 fs_manager.cc:359] Metadata directory not provided
I20260808 07:02:30.719044 19665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:02:30.719204 19665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:02:30.720247 19665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "39f0b8ec22d44894b20ce12b523a6a8e"
format_stamp: "Formatted at 2026-08-08 07:02:30 on dist-test-slave-0b58"
I20260808 07:02:30.720712 19665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "39f0b8ec22d44894b20ce12b523a6a8e"
format_stamp: "Formatted at 2026-08-08 07:02:30 on dist-test-slave-0b58"
I20260808 07:02:30.722112 19665 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 07:02:30.723004 19681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:02:30.723187 19665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:02:30.723306 19665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
uuid: "39f0b8ec22d44894b20ce12b523a6a8e"
format_stamp: "Formatted at 2026-08-08 07:02:30 on dist-test-slave-0b58"
I20260808 07:02:30.723418 19665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:02:30.744493 19665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:02:30.744863 19665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:02:30.745030 19665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:02:30.749303 19665 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.190:43327
I20260808 07:02:30.749347 19739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.190:43327 every 8 connection(s)
I20260808 07:02:30.749714 19665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260808 07:02:30.750456 19740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:02:30.753085 19740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39f0b8ec22d44894b20ce12b523a6a8e: Bootstrap starting.
I20260808 07:02:30.753667 19740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39f0b8ec22d44894b20ce12b523a6a8e: Neither blocks nor log segments found. Creating new log.
I20260808 07:02:30.753931 19740 log.cc:824] T 00000000000000000000000000000000 P 39f0b8ec22d44894b20ce12b523a6a8e: Log is configured to *not* fsync() on all Append() calls
I20260808 07:02:30.754558 19740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39f0b8ec22d44894b20ce12b523a6a8e: No bootstrap required, opened a new log
I20260808 07:02:30.756151 19740 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39f0b8ec22d44894b20ce12b523a6a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39f0b8ec22d44894b20ce12b523a6a8e" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 43327 } }
I20260808 07:02:30.756397 19740 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39f0b8ec22d44894b20ce12b523a6a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:02:30.756428 19740 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39f0b8ec22d44894b20ce12b523a6a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0b8ec22d44894b20ce12b523a6a8e, State: Initialized, Role: FOLLOWER
I20260808 07:02:30.756574 19740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39f0b8ec22d44894b20ce12b523a6a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39f0b8ec22d44894b20ce12b523a6a8e" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 43327 } }
I20260808 07:02:30.756665 19740 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39f0b8ec22d44894b20ce12b523a6a8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:02:30.756690 19740 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39f0b8ec22d44894b20ce12b523a6a8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:02:30.756719 19740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39f0b8ec22d44894b20ce12b523a6a8e [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:02:30.757236 19740 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39f0b8ec22d44894b20ce12b523a6a8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39f0b8ec22d44894b20ce12b523a6a8e" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 43327 } }
I20260808 07:02:30.757341 19740 leader_election.cc:302] T 00000000000000000000000000000000 P 39f0b8ec22d44894b20ce12b523a6a8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39f0b8ec22d44894b20ce12b523a6a8e; no voters:
I20260808 07:02:30.757527 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 19665
I20260808 07:02:30.757575 19740 leader_election.cc:288] T 00000000000000000000000000000000 P 39f0b8ec22d44894b20ce12b523a6a8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:02:30.757637 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260808 07:02:30.757647 19743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39f0b8ec22d44894b20ce12b523a6a8e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:02:30.757859 19743 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39f0b8ec22d44894b20ce12b523a6a8e [term 1 LEADER]: Becoming Leader. State: Replica: 39f0b8ec22d44894b20ce12b523a6a8e, State: Running, Role: LEADER
I20260808 07:02:30.757920 19740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39f0b8ec22d44894b20ce12b523a6a8e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:02:30.758041 19743 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39f0b8ec22d44894b20ce12b523a6a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39f0b8ec22d44894b20ce12b523a6a8e" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 43327 } }
I20260808 07:02:30.758528 19744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39f0b8ec22d44894b20ce12b523a6a8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39f0b8ec22d44894b20ce12b523a6a8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39f0b8ec22d44894b20ce12b523a6a8e" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 43327 } } }
I20260808 07:02:30.758626 19744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39f0b8ec22d44894b20ce12b523a6a8e [sys.catalog]: This master's current role is: LEADER
I20260808 07:02:30.758790 19745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39f0b8ec22d44894b20ce12b523a6a8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39f0b8ec22d44894b20ce12b523a6a8e. Latest consensus state: current_term: 1 leader_uuid: "39f0b8ec22d44894b20ce12b523a6a8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39f0b8ec22d44894b20ce12b523a6a8e" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 43327 } } }
I20260808 07:02:30.758881 19745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39f0b8ec22d44894b20ce12b523a6a8e [sys.catalog]: This master's current role is: LEADER
I20260808 07:02:30.759042 19754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:02:30.760129 19754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:02:30.761910 19754 catalog_manager.cc:1347] Generated new cluster ID: fcd44b938c0c4a17b72868124b90284f
I20260808 07:02:30.761976 19754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:02:30.794176 19754 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:02:30.794905 19754 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:02:30.801100 19754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39f0b8ec22d44894b20ce12b523a6a8e: Generated new TSK 0
I20260808 07:02:30.801309 19754 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:02:30.804351 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.129:0
--local_ip_for_outbound_sockets=127.15.250.129
--webserver_interface=127.15.250.129
--webserver_port=0
--tserver_master_addrs=127.15.250.190:43327
--builtin_ntp_servers=127.15.250.148:39425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
W20260808 07:02:30.890525 19764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:02:30.890787 19764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:02:30.890828 19764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:02:30.892040 19764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:02:30.892132 19764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.129
I20260808 07:02:30.893559 19764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:39425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.250.129
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.15.250.190:43327
--never_fsync=true
--heap_profile_path=/tmp/kudu.19764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.129
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:02:30.893869 19764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:02:30.894196 19764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:02:30.900425 19772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:02:30.900426 19769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:02:30.900413 19770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:02:30.900797 19764 server_base.cc:1034] running on GCE node
I20260808 07:02:30.901038 19764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:02:30.901346 19764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:02:30.902513 19764 hybrid_clock.cc:648] HybridClock initialized: now 1786172550902500 us; error 26 us; skew 500 ppm
I20260808 07:02:30.903843 19764 webserver.cc:460] Webserver started at http://127.15.250.129:34641/ using document root <none> and password file <none>
I20260808 07:02:30.904050 19764 fs_manager.cc:359] Metadata directory not provided
I20260808 07:02:30.904093 19764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:02:30.904186 19764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:02:30.905272 19764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d20f18f45fb644f3a9741e9dc33f4dea"
format_stamp: "Formatted at 2026-08-08 07:02:30 on dist-test-slave-0b58"
I20260808 07:02:30.905581 19764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d20f18f45fb644f3a9741e9dc33f4dea"
format_stamp: "Formatted at 2026-08-08 07:02:30 on dist-test-slave-0b58"
I20260808 07:02:30.906991 19764 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 07:02:30.907725 19778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:02:30.907907 19764 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 07:02:30.908000 19764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d20f18f45fb644f3a9741e9dc33f4dea"
format_stamp: "Formatted at 2026-08-08 07:02:30 on dist-test-slave-0b58"
I20260808 07:02:30.908061 19764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:02:30.922848 19764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:02:30.923157 19764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:02:30.923265 19764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:02:30.923468 19764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:02:30.923838 19764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:02:30.923883 19764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:02:30.923907 19764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:02:30.923955 19764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:02:30.930020 19764 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.129:38229
I20260808 07:02:30.930119 19901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.129:38229 every 8 connection(s)
I20260808 07:02:30.930511 19764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260808 07:02:30.936589 19902 heartbeater.cc:347] Connected to a master server at 127.15.250.190:43327
I20260808 07:02:30.936697 19902 heartbeater.cc:464] Registering TS with master...
I20260808 07:02:30.936852 19902 heartbeater.cc:511] Master 127.15.250.190:43327 requested a full tablet report, sending...
I20260808 07:02:30.937366 19699 ts_manager.cc:194] Registered new tserver with Master: d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229)
I20260808 07:02:30.937964 19699 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.129:53359
I20260808 07:02:30.940694 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 19764
I20260808 07:02:30.940800 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260808 07:02:30.941957 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.130:0
--local_ip_for_outbound_sockets=127.15.250.130
--webserver_interface=127.15.250.130
--webserver_port=0
--tserver_master_addrs=127.15.250.190:43327
--builtin_ntp_servers=127.15.250.148:39425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
W20260808 07:02:31.028316 19905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:02:31.028577 19905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:02:31.028620 19905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:02:31.029876 19905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:02:31.029978 19905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.130
I20260808 07:02:31.031358 19905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:39425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.250.130
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.15.250.190:43327
--never_fsync=true
--heap_profile_path=/tmp/kudu.19905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.130
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:02:31.031641 19905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:02:31.031973 19905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:02:31.037582 19910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:02:31.037580 19913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:02:31.037563 19911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:02:31.037886 19905 server_base.cc:1034] running on GCE node
I20260808 07:02:31.038096 19905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:02:31.038357 19905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:02:31.039527 19905 hybrid_clock.cc:648] HybridClock initialized: now 1786172551039510 us; error 29 us; skew 500 ppm
I20260808 07:02:31.040961 19905 webserver.cc:460] Webserver started at http://127.15.250.130:42867/ using document root <none> and password file <none>
I20260808 07:02:31.041251 19905 fs_manager.cc:359] Metadata directory not provided
I20260808 07:02:31.041332 19905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:02:31.041486 19905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:02:31.042514 19905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "886bd7a0d91c4ca6947754a55fa13bf8"
format_stamp: "Formatted at 2026-08-08 07:02:31 on dist-test-slave-0b58"
I20260808 07:02:31.042898 19905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "886bd7a0d91c4ca6947754a55fa13bf8"
format_stamp: "Formatted at 2026-08-08 07:02:31 on dist-test-slave-0b58"
I20260808 07:02:31.044328 19905 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260808 07:02:31.045128 19920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:02:31.045451 19905 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 07:02:31.045565 19905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "886bd7a0d91c4ca6947754a55fa13bf8"
format_stamp: "Formatted at 2026-08-08 07:02:31 on dist-test-slave-0b58"
I20260808 07:02:31.045634 19905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:02:31.060786 19905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:02:31.061141 19905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:02:31.061304 19905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:02:31.061544 19905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:02:31.061887 19905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:02:31.061944 19905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:02:31.061987 19905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:02:31.062016 19905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:02:31.068192 19905 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.130:43897
I20260808 07:02:31.068244 20041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.130:43897 every 8 connection(s)
I20260808 07:02:31.068646 19905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260808 07:02:31.074663 20042 heartbeater.cc:347] Connected to a master server at 127.15.250.190:43327
I20260808 07:02:31.074796 20042 heartbeater.cc:464] Registering TS with master...
I20260808 07:02:31.075014 20042 heartbeater.cc:511] Master 127.15.250.190:43327 requested a full tablet report, sending...
I20260808 07:02:31.075389 19699 ts_manager.cc:194] Registered new tserver with Master: 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897)
I20260808 07:02:31.075747 19699 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.130:43587
I20260808 07:02:31.077958 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 19905
I20260808 07:02:31.078050 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260808 07:02:31.079329 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.131:0
--local_ip_for_outbound_sockets=127.15.250.131
--webserver_interface=127.15.250.131
--webserver_port=0
--tserver_master_addrs=127.15.250.190:43327
--builtin_ntp_servers=127.15.250.148:39425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
W20260808 07:02:31.165001 20045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:02:31.165262 20045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:02:31.165287 20045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:02:31.166887 20045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:02:31.166989 20045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.131
I20260808 07:02:31.168762 20045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:39425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.250.131
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.15.250.190:43327
--never_fsync=true
--heap_profile_path=/tmp/kudu.20045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.131
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:02:31.169100 20045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:02:31.169492 20045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:02:31.175118 20053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:02:31.175073 20051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:02:31.175093 20050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:02:31.175463 20045 server_base.cc:1034] running on GCE node
I20260808 07:02:31.175645 20045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:02:31.175861 20045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:02:31.177034 20045 hybrid_clock.cc:648] HybridClock initialized: now 1786172551177018 us; error 29 us; skew 500 ppm
I20260808 07:02:31.178460 20045 webserver.cc:460] Webserver started at http://127.15.250.131:44603/ using document root <none> and password file <none>
I20260808 07:02:31.178700 20045 fs_manager.cc:359] Metadata directory not provided
I20260808 07:02:31.178747 20045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:02:31.178854 20045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:02:31.179855 20045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "9b54f999704c45bca763910c6647c87a"
format_stamp: "Formatted at 2026-08-08 07:02:31 on dist-test-slave-0b58"
I20260808 07:02:31.180218 20045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "9b54f999704c45bca763910c6647c87a"
format_stamp: "Formatted at 2026-08-08 07:02:31 on dist-test-slave-0b58"
I20260808 07:02:31.181674 20045 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 07:02:31.182478 20059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:02:31.182682 20045 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 07:02:31.182760 20045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9b54f999704c45bca763910c6647c87a"
format_stamp: "Formatted at 2026-08-08 07:02:31 on dist-test-slave-0b58"
I20260808 07:02:31.182817 20045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:02:31.202652 20045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:02:31.202952 20045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:02:31.203058 20045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:02:31.203262 20045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:02:31.203644 20045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:02:31.203689 20045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:02:31.203738 20045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:02:31.203765 20045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:02:31.210151 20045 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.131:44861
I20260808 07:02:31.210340 20181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.131:44861 every 8 connection(s)
I20260808 07:02:31.210641 20045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260808 07:02:31.215788 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 20045
I20260808 07:02:31.215912 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260808 07:02:31.217108 20182 heartbeater.cc:347] Connected to a master server at 127.15.250.190:43327
I20260808 07:02:31.217240 20182 heartbeater.cc:464] Registering TS with master...
I20260808 07:02:31.217430 20182 heartbeater.cc:511] Master 127.15.250.190:43327 requested a full tablet report, sending...
I20260808 07:02:31.217824 19699 ts_manager.cc:194] Registered new tserver with Master: 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:31.218212 19699 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.131:57725
I20260808 07:02:31.228201 16362 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 07:02:31.229329 16362 test_util.cc:268] Using random seed: -1267961970
I20260808 07:02:31.236800 19699 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35120:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 07:02:31.237149 19699 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 07:02:31.244731 19970 tablet_service.cc:1467] Processing CreateTablet for tablet edea628254264454bbd00d793271a973 (DEFAULT_TABLE table=test-workload [id=10654ea2be7140858c4039a5f4d3639f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:02:31.244835 19830 tablet_service.cc:1467] Processing CreateTablet for tablet edea628254264454bbd00d793271a973 (DEFAULT_TABLE table=test-workload [id=10654ea2be7140858c4039a5f4d3639f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:02:31.245194 19970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edea628254264454bbd00d793271a973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:02:31.245283 19830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edea628254264454bbd00d793271a973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:02:31.247891 20197 tablet_bootstrap.cc:492] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8: Bootstrap starting.
I20260808 07:02:31.248009 20198 tablet_bootstrap.cc:492] T edea628254264454bbd00d793271a973 P d20f18f45fb644f3a9741e9dc33f4dea: Bootstrap starting.
I20260808 07:02:31.248667 20197 tablet_bootstrap.cc:654] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8: Neither blocks nor log segments found. Creating new log.
I20260808 07:02:31.248732 20198 tablet_bootstrap.cc:654] T edea628254264454bbd00d793271a973 P d20f18f45fb644f3a9741e9dc33f4dea: Neither blocks nor log segments found. Creating new log.
I20260808 07:02:31.248624 20112 tablet_service.cc:1467] Processing CreateTablet for tablet edea628254264454bbd00d793271a973 (DEFAULT_TABLE table=test-workload [id=10654ea2be7140858c4039a5f4d3639f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:02:31.249022 20112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edea628254264454bbd00d793271a973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:02:31.249035 20198 log.cc:824] T edea628254264454bbd00d793271a973 P d20f18f45fb644f3a9741e9dc33f4dea: Log is configured to *not* fsync() on all Append() calls
I20260808 07:02:31.249048 20197 log.cc:824] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8: Log is configured to *not* fsync() on all Append() calls
I20260808 07:02:31.249801 20198 tablet_bootstrap.cc:492] T edea628254264454bbd00d793271a973 P d20f18f45fb644f3a9741e9dc33f4dea: No bootstrap required, opened a new log
I20260808 07:02:31.249890 20198 ts_tablet_manager.cc:1397] T edea628254264454bbd00d793271a973 P d20f18f45fb644f3a9741e9dc33f4dea: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:02:31.250029 20197 tablet_bootstrap.cc:492] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8: No bootstrap required, opened a new log
I20260808 07:02:31.250104 20197 ts_tablet_manager.cc:1397] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:02:31.252019 20198 raft_consensus.cc:348] T edea628254264454bbd00d793271a973 P d20f18f45fb644f3a9741e9dc33f4dea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:31.252185 20198 raft_consensus.cc:374] T edea628254264454bbd00d793271a973 P d20f18f45fb644f3a9741e9dc33f4dea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:02:31.252233 20198 raft_consensus.cc:729] T edea628254264454bbd00d793271a973 P d20f18f45fb644f3a9741e9dc33f4dea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f18f45fb644f3a9741e9dc33f4dea, State: Initialized, Role: FOLLOWER
I20260808 07:02:31.252442 20198 consensus_queue.cc:260] T edea628254264454bbd00d793271a973 P d20f18f45fb644f3a9741e9dc33f4dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:31.252594 20201 tablet_bootstrap.cc:492] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a: Bootstrap starting.
I20260808 07:02:31.252737 20198 ts_tablet_manager.cc:1428] T edea628254264454bbd00d793271a973 P d20f18f45fb644f3a9741e9dc33f4dea: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260808 07:02:31.252890 19902 heartbeater.cc:503] Master 127.15.250.190:43327 was elected leader, sending a full tablet report...
I20260808 07:02:31.252951 20204 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P d20f18f45fb644f3a9741e9dc33f4dea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 07:02:31.252871 20197 raft_consensus.cc:348] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:31.252998 20204 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P d20f18f45fb644f3a9741e9dc33f4dea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:31.253033 20197 raft_consensus.cc:374] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:02:31.253078 20197 raft_consensus.cc:729] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 886bd7a0d91c4ca6947754a55fa13bf8, State: Initialized, Role: FOLLOWER
I20260808 07:02:31.253219 20197 consensus_queue.cc:260] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:31.253294 20201 tablet_bootstrap.cc:654] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a: Neither blocks nor log segments found. Creating new log.
I20260808 07:02:31.253515 20197 ts_tablet_manager.cc:1428] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260808 07:02:31.253659 20042 heartbeater.cc:503] Master 127.15.250.190:43327 was elected leader, sending a full tablet report...
I20260808 07:02:31.253804 20201 log.cc:824] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a: Log is configured to *not* fsync() on all Append() calls
I20260808 07:02:31.254544 20201 tablet_bootstrap.cc:492] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a: No bootstrap required, opened a new log
I20260808 07:02:31.254618 20201 ts_tablet_manager.cc:1397] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 07:02:31.254748 20205 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 07:02:31.254827 20205 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:31.255080 20205 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:31.256695 20201 raft_consensus.cc:348] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:31.256872 20201 raft_consensus.cc:374] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:02:31.256910 20201 raft_consensus.cc:729] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b54f999704c45bca763910c6647c87a, State: Initialized, Role: FOLLOWER
I20260808 07:02:31.257028 20201 consensus_queue.cc:260] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:31.257313 20201 ts_tablet_manager.cc:1428] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 07:02:31.257349 19992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" is_pre_election: true
I20260808 07:02:31.257443 19992 raft_consensus.cc:2454] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f18f45fb644f3a9741e9dc33f4dea in term 0.
I20260808 07:02:31.257616 20204 leader_election.cc:288] T edea628254264454bbd00d793271a973 P d20f18f45fb644f3a9741e9dc33f4dea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:31.257690 19781 leader_election.cc:302] T edea628254264454bbd00d793271a973 P d20f18f45fb644f3a9741e9dc33f4dea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8, d20f18f45fb644f3a9741e9dc33f4dea; no voters:
I20260808 07:02:31.257980 20182 heartbeater.cc:503] Master 127.15.250.190:43327 was elected leader, sending a full tablet report...
I20260808 07:02:31.258284 20204 raft_consensus.cc:2781] T edea628254264454bbd00d793271a973 P d20f18f45fb644f3a9741e9dc33f4dea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 07:02:31.258350 20204 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P d20f18f45fb644f3a9741e9dc33f4dea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 07:02:31.258374 20204 raft_consensus.cc:3037] T edea628254264454bbd00d793271a973 P d20f18f45fb644f3a9741e9dc33f4dea [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:02:31.259290 20204 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P d20f18f45fb644f3a9741e9dc33f4dea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:31.259516 20204 leader_election.cc:288] T edea628254264454bbd00d793271a973 P d20f18f45fb644f3a9741e9dc33f4dea [CANDIDATE]: Term 1 election: Requested vote from peers 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:31.259569 19992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886bd7a0d91c4ca6947754a55fa13bf8"
I20260808 07:02:31.259645 19992 raft_consensus.cc:3037] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:02:31.260141 20133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:02:31.260342 20133 raft_consensus.cc:2454] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 886bd7a0d91c4ca6947754a55fa13bf8 in term 0.
I20260808 07:02:31.260509 19992 raft_consensus.cc:2454] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f18f45fb644f3a9741e9dc33f4dea in term 1.
I20260808 07:02:31.260543 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8, 9b54f999704c45bca763910c6647c87a; no voters:
I20260808 07:02:31.260679 20205 raft_consensus.cc:2741] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260808 07:02:31.260686 19781 leader_election.cc:302] T edea628254264454bbd00d793271a973 P d20f18f45fb644f3a9741e9dc33f4dea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8, d20f18f45fb644f3a9741e9dc33f4dea; no voters:
I20260808 07:02:31.260730 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:31.260834 19851 raft_consensus.cc:2379] T edea628254264454bbd00d793271a973 P d20f18f45fb644f3a9741e9dc33f4dea [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 886bd7a0d91c4ca6947754a55fa13bf8 in current term 1: Already voted for candidate d20f18f45fb644f3a9741e9dc33f4dea in this term.
I20260808 07:02:31.260922 20203 raft_consensus.cc:2781] T edea628254264454bbd00d793271a973 P d20f18f45fb644f3a9741e9dc33f4dea [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:02:31.260978 20210 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 07:02:31.261032 20210 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:31.261168 20203 raft_consensus.cc:686] T edea628254264454bbd00d793271a973 P d20f18f45fb644f3a9741e9dc33f4dea [term 1 LEADER]: Becoming Leader. State: Replica: d20f18f45fb644f3a9741e9dc33f4dea, State: Running, Role: LEADER
I20260808 07:02:31.261348 20210 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897)
I20260808 07:02:31.261490 20203 consensus_queue.cc:237] T edea628254264454bbd00d793271a973 P d20f18f45fb644f3a9741e9dc33f4dea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:31.262879 19699 catalog_manager.cc:5495] T edea628254264454bbd00d793271a973 P d20f18f45fb644f3a9741e9dc33f4dea reported cstate change: term changed from 0 to 1, leader changed from <none> to d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129). New cstate: current_term: 1 leader_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } health_report { overall_health: UNKNOWN } } }
I20260808 07:02:31.263135 20133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:02:31.263127 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a"
I20260808 07:02:31.263242 20133 raft_consensus.cc:2454] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f18f45fb644f3a9741e9dc33f4dea in term 0.
I20260808 07:02:31.264312 20132 raft_consensus.cc:3037] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:02:31.265079 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:31.265331 20203 consensus_queue.cc:1035] T edea628254264454bbd00d793271a973 P d20f18f45fb644f3a9741e9dc33f4dea [LEADER]: Connected to new peer: Peer: permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:02:31.265527 20204 consensus_queue.cc:1035] T edea628254264454bbd00d793271a973 P d20f18f45fb644f3a9741e9dc33f4dea [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 07:02:31.267822 20062 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.15.250.129:38229 timed out after 0.006s (SENT)
I20260808 07:02:31.268999 19992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" is_pre_election: true
I20260808 07:02:31.269332 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:31.269526 20062 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b54f999704c45bca763910c6647c87a; no voters: 886bd7a0d91c4ca6947754a55fa13bf8, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:31.269678 20208 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 07:02:31.276540 20217 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260808 07:02:31.276600 20217 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20260808 07:02:31.586527 20132 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260808 07:02:31.586578 19991 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260808 07:02:31.586622 20132 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20260808 07:02:31.586643 19991 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260808 07:02:32.145539 20208 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:32.145679 20208 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:32.146077 20208 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897)
I20260808 07:02:32.146324 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:32.146384 19992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" is_pre_election: true
I20260808 07:02:32.146683 20062 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b54f999704c45bca763910c6647c87a; no voters: 886bd7a0d91c4ca6947754a55fa13bf8, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:32.146821 20208 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:32.288518 20205 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:32.288622 20205 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:32.288924 20205 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:32.289119 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:32.289130 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:02:32.289434 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8; no voters: 9b54f999704c45bca763910c6647c87a, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:32.289578 20205 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:33.156060 20208 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:33.156167 20208 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:33.156409 20208 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897)
I20260808 07:02:33.156739 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:33.156749 19992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" is_pre_election: true
I20260808 07:02:33.157065 20062 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b54f999704c45bca763910c6647c87a; no voters: 886bd7a0d91c4ca6947754a55fa13bf8, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:33.157191 20208 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:33.157274 20240 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=1, attempt_no=0}) took 1881 ms. Trace:
I20260808 07:02:33.157418 20240 rpcz_store.cc:276] 0808 07:02:31.275416 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:31.275459 (+ 43us) service_pool.cc:227] Handling call
0808 07:02:33.157265 (+1881806us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:31.586031 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:31.586070 (+ 39us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:31.586070 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:31.586071 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:31.586083 (+ 12us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:31.586093 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:31.586094 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:31.586104 (+ 10us) tablet.cc:699] Row locks acquired
0808 07:02:31.586105 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:02:31.586112 (+ 7us) write_op.cc:270] Start()
0808 07:02:32.336230 (+750118us) write_op.cc:276] Timestamp: P: 1786172551586088 usec, L: 0
0808 07:02:32.336232 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:02:32.336313 (+ 81us) log.cc:842] Serialized 345 byte log entry
0808 07:02:33.156993 (+820680us) op_driver.cc:444] REPLICATION: finished
0808 07:02:33.157093 (+ 100us) write_op.cc:301] APPLY: starting
0808 07:02:33.157099 (+ 6us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:33.157103 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:33.157105 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:33.157120 (+ 15us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:33.157126 (+ 6us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:33.157126 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:02:33.157146 (+ 20us) log.cc:842] Serialized 22 byte log entry
0808 07:02:33.157238 (+ 92us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:33.157242 (+ 4us) write_op.cc:454] Released schema lock
0808 07:02:33.157243 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":97920,"child_traces":[["op",{"apply.queue_time_us":90,"mutex_wait_us":34,"num_ops":1,"prepare.queue_time_us":309848,"prepare.run_cpu_time_us":248,"prepare.run_wall_time_us":750315,"raft.queue_time_us":15,"raft.run_cpu_time_us":41,"raft.run_wall_time_us":41,"replication_time_us":820757,"thread_start_us":61,"threads_started":1}]]}
I20260808 07:02:33.231287 20205 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:33.231395 20205 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:33.231647 20205 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:33.232018 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:33.233170 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:02:33.233417 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8; no voters: 9b54f999704c45bca763910c6647c87a, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:33.233577 20205 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:33.486370 20208 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:33.486485 20208 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:33.486680 20208 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897)
I20260808 07:02:33.486922 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:33.487040 19992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" is_pre_election: true
I20260808 07:02:33.487159 20240 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=2, attempt_no=0}) took 1339 ms. Trace:
I20260808 07:02:33.487219 20240 rpcz_store.cc:276] 0808 07:02:32.147256 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:32.147302 (+ 46us) service_pool.cc:227] Handling call
0808 07:02:33.487150 (+1339848us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:32.336356 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:32.336371 (+ 15us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:32.336372 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:32.336372 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:32.336398 (+ 26us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:32.336401 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:32.336403 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:32.336412 (+ 9us) tablet.cc:699] Row locks acquired
0808 07:02:32.336413 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:02:32.336421 (+ 8us) write_op.cc:270] Start()
0808 07:02:33.296554 (+960133us) write_op.cc:276] Timestamp: P: 1786172552336375 usec, L: 0
0808 07:02:33.296555 (+ 1us) op_driver.cc:331] REPLICATION: starting
0808 07:02:33.296612 (+ 57us) log.cc:842] Serialized 354 byte log entry
0808 07:02:33.486955 (+190343us) op_driver.cc:444] REPLICATION: finished
0808 07:02:33.486989 (+ 34us) write_op.cc:301] APPLY: starting
0808 07:02:33.486994 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:33.486998 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:33.487001 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:33.487016 (+ 15us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:33.487021 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:33.487022 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:33.487044 (+ 22us) log.cc:842] Serialized 22 byte log entry
0808 07:02:33.487125 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:33.487128 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:33.487128 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":20,"num_ops":1,"prepare.queue_time_us":188847,"prepare.run_cpu_time_us":243,"prepare.run_wall_time_us":960335,"raft.queue_time_us":118,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":98,"replication_time_us":190395,"spinlock_wait_cycles":4480}]]}
I20260808 07:02:33.487319 20062 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b54f999704c45bca763910c6647c87a; no voters: 886bd7a0d91c4ca6947754a55fa13bf8, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:33.487401 20208 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:33.909469 20205 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:33.909574 20205 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:33.909792 20205 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:33.910003 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:33.910243 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:02:33.910501 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8; no voters: 9b54f999704c45bca763910c6647c87a, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:33.910647 20205 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:34.377656 20247 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:34.377760 20247 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:34.377987 20247 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897)
I20260808 07:02:34.378185 19992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" is_pre_election: true
I20260808 07:02:34.378217 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:34.378440 20062 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b54f999704c45bca763910c6647c87a; no voters: 886bd7a0d91c4ca6947754a55fa13bf8, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:34.378886 20249 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=3, attempt_no=0}) took 1220 ms. Trace:
I20260808 07:02:34.378964 20249 rpcz_store.cc:276] 0808 07:02:33.158182 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:33.158203 (+ 21us) service_pool.cc:227] Handling call
0808 07:02:34.378872 (+1220669us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:33.296699 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:33.296715 (+ 16us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:33.296716 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:33.296717 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:33.296733 (+ 16us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:33.296737 (+ 4us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:33.296738 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:33.296751 (+ 13us) tablet.cc:699] Row locks acquired
0808 07:02:33.296752 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:02:33.296760 (+ 8us) write_op.cc:270] Start()
0808 07:02:34.168869 (+872109us) write_op.cc:276] Timestamp: P: 1786172553296721 usec, L: 0
0808 07:02:34.168871 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:02:34.168919 (+ 48us) log.cc:842] Serialized 354 byte log entry
0808 07:02:34.378033 (+209114us) op_driver.cc:444] REPLICATION: finished
0808 07:02:34.378157 (+ 124us) write_op.cc:301] APPLY: starting
0808 07:02:34.378162 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:34.378167 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:34.378169 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:34.378235 (+ 66us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:34.378240 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:34.378241 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:34.378336 (+ 95us) log.cc:842] Serialized 43 byte log entry
0808 07:02:34.378417 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:34.378420 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:34.378421 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":111,"mutex_wait_us":25,"num_ops":1,"prepare.queue_time_us":138324,"prepare.run_cpu_time_us":279,"prepare.run_wall_time_us":872369,"raft.queue_time_us":153,"raft.run_cpu_time_us":31,"raft.run_wall_time_us":31,"replication_time_us":209158,"thread_start_us":124,"threads_started":2}]]}
I20260808 07:02:34.617928 20247 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:34.618495 20249 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=4, attempt_no=0}) took 1130 ms. Trace:
I20260808 07:02:34.618582 20249 rpcz_store.cc:276] 0808 07:02:33.487962 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:33.488011 (+ 49us) service_pool.cc:227] Handling call
0808 07:02:34.618480 (+1130469us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:34.169078 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:34.169095 (+ 17us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:34.169096 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:34.169096 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:34.169109 (+ 13us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:34.169112 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:34.169114 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:34.169124 (+ 10us) tablet.cc:699] Row locks acquired
0808 07:02:34.169124 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:34.169132 (+ 8us) write_op.cc:270] Start()
0808 07:02:34.224304 (+ 55172us) write_op.cc:276] Timestamp: P: 1786172554169093 usec, L: 0
0808 07:02:34.224307 (+ 3us) op_driver.cc:331] REPLICATION: starting
0808 07:02:34.224411 (+ 104us) log.cc:842] Serialized 354 byte log entry
0808 07:02:34.618287 (+393876us) op_driver.cc:444] REPLICATION: finished
0808 07:02:34.618356 (+ 69us) write_op.cc:301] APPLY: starting
0808 07:02:34.618362 (+ 6us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:34.618369 (+ 7us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:34.618372 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:34.618393 (+ 21us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:34.618398 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:34.618399 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:34.618427 (+ 28us) log.cc:842] Serialized 43 byte log entry
0808 07:02:34.618448 (+ 21us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:34.618451 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:34.618451 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":52,"num_ops":1,"prepare.queue_time_us":680941,"prepare.run_cpu_time_us":238,"prepare.run_wall_time_us":55373,"replication_time_us":393975}]]}
I20260808 07:02:34.930804 20205 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:34.930912 20205 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:34.931119 20205 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:34.931371 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:02:34.931397 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:34.931690 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8; no voters: 9b54f999704c45bca763910c6647c87a, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:35.342868 20247 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:35.343000 20247 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:35.343236 20247 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897)
I20260808 07:02:35.343508 19992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" is_pre_election: true
I20260808 07:02:35.343534 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:35.343813 20062 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b54f999704c45bca763910c6647c87a; no voters: 886bd7a0d91c4ca6947754a55fa13bf8, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:35.343982 20247 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:35.763545 20205 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:35.868397 20205 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:35.868513 20205 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:35.868762 20205 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:35.868996 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:02:35.869055 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:35.869310 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8; no voters: 9b54f999704c45bca763910c6647c87a, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:35.869441 20205 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:36.076921 20205 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:36.077052 20205 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:36.077288 20205 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:36.077545 20133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:02:36.077543 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:36.077656 20260 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=6, attempt_no=0}) took 1458 ms. Trace:
I20260808 07:02:36.077695 20260 rpcz_store.cc:276] 0808 07:02:34.619241 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:34.619272 (+ 31us) service_pool.cc:227] Handling call
0808 07:02:36.077642 (+1458370us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:35.110062 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:35.110075 (+ 13us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:35.110075 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:35.110076 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:35.110086 (+ 10us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:35.110089 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:35.110090 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:35.110098 (+ 8us) tablet.cc:699] Row locks acquired
0808 07:02:35.110099 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:02:35.110105 (+ 6us) write_op.cc:270] Start()
0808 07:02:36.015197 (+905092us) write_op.cc:276] Timestamp: P: 1786172555110060 usec, L: 0
0808 07:02:36.015199 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:02:36.015254 (+ 55us) log.cc:842] Serialized 354 byte log entry
0808 07:02:36.077368 (+ 62114us) op_driver.cc:444] REPLICATION: finished
0808 07:02:36.077521 (+ 153us) write_op.cc:301] APPLY: starting
0808 07:02:36.077525 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:36.077529 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:36.077532 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:36.077544 (+ 12us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:36.077549 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:36.077550 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:36.077570 (+ 20us) log.cc:842] Serialized 22 byte log entry
0808 07:02:36.077618 (+ 48us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:36.077620 (+ 2us) write_op.cc:454] Released schema lock
0808 07:02:36.077621 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":141,"num_ops":1,"prepare.queue_time_us":490676,"prepare.run_cpu_time_us":158,"prepare.run_wall_time_us":905223,"raft.queue_time_us":19,"raft.run_cpu_time_us":42,"raft.run_wall_time_us":43,"replication_time_us":62164,"thread_start_us":76,"threads_started":1}]]}
I20260808 07:02:36.077973 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8; no voters: 9b54f999704c45bca763910c6647c87a, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:36.078079 20205 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:36.318095 20259 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:36.318267 20259 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:36.318470 20259 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897)
I20260808 07:02:36.318688 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:36.318706 19991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" is_pre_election: true
I20260808 07:02:36.319018 20062 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b54f999704c45bca763910c6647c87a; no voters: 886bd7a0d91c4ca6947754a55fa13bf8, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:36.319115 20259 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:37.630501 20264 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:37.630717 20264 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:37.630983 20264 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:37.631208 20133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:02:37.631239 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:37.631351 20270 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=7, attempt_no=0}) took 2287 ms. Trace:
I20260808 07:02:37.631402 20270 rpcz_store.cc:276] 0808 07:02:35.344282 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:35.344379 (+ 97us) service_pool.cc:227] Handling call
0808 07:02:37.631343 (+2286964us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:36.015295 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:36.015312 (+ 17us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:36.015313 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:36.015313 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:36.015325 (+ 12us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:36.015328 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:36.015329 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:36.015378 (+ 49us) tablet.cc:699] Row locks acquired
0808 07:02:36.015378 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:36.015388 (+ 10us) write_op.cc:270] Start()
0808 07:02:36.981481 (+966093us) write_op.cc:276] Timestamp: P: 1786172556015320 usec, L: 0
0808 07:02:36.981483 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:02:36.981544 (+ 61us) log.cc:842] Serialized 354 byte log entry
0808 07:02:37.631149 (+649605us) op_driver.cc:444] REPLICATION: finished
0808 07:02:37.631248 (+ 99us) write_op.cc:301] APPLY: starting
0808 07:02:37.631252 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:37.631257 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:37.631260 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:37.631273 (+ 13us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:37.631279 (+ 6us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:37.631280 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:37.631298 (+ 18us) log.cc:842] Serialized 22 byte log entry
0808 07:02:37.631320 (+ 22us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:37.631323 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:37.631324 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":89,"num_ops":1,"prepare.queue_time_us":670809,"prepare.run_cpu_time_us":214,"prepare.run_wall_time_us":966273,"replication_time_us":649661,"thread_start_us":39,"threads_started":1}]]}
I20260808 07:02:37.631494 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8; no voters: 9b54f999704c45bca763910c6647c87a, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:37.631601 20264 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:37.971339 20263 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:37.971454 20263 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:37.971660 20263 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897)
I20260808 07:02:37.971868 19992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" is_pre_election: true
I20260808 07:02:37.971870 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:37.972186 20062 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b54f999704c45bca763910c6647c87a; no voters: 886bd7a0d91c4ca6947754a55fa13bf8, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:38.180388 20264 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:38.180512 20264 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:38.180773 20264 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:38.180989 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:38.181015 20133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:02:38.181274 20279 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=8, attempt_no=0}) took 2103 ms. Trace:
I20260808 07:02:38.181352 20279 rpcz_store.cc:276] 0808 07:02:36.078245 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:36.078274 (+ 29us) service_pool.cc:227] Handling call
0808 07:02:38.181264 (+2102990us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:36.981576 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:36.981592 (+ 16us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:36.981593 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:36.981593 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:36.981605 (+ 12us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:36.981608 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:36.981609 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:36.981620 (+ 11us) tablet.cc:699] Row locks acquired
0808 07:02:36.981621 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:02:36.981629 (+ 8us) write_op.cc:270] Start()
0808 07:02:37.046741 (+ 65112us) write_op.cc:276] Timestamp: P: 1786172556981564 usec, L: 0
0808 07:02:37.046743 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:02:37.046813 (+ 70us) log.cc:842] Serialized 354 byte log entry
0808 07:02:38.181017 (+1134204us) op_driver.cc:444] REPLICATION: finished
0808 07:02:38.181107 (+ 90us) write_op.cc:301] APPLY: starting
0808 07:02:38.181111 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:38.181115 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:38.181119 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:38.181135 (+ 16us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:38.181140 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:38.181141 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:38.181159 (+ 18us) log.cc:842] Serialized 22 byte log entry
0808 07:02:38.181239 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:38.181243 (+ 4us) write_op.cc:454] Released schema lock
0808 07:02:38.181243 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":81,"num_ops":1,"prepare.queue_time_us":903190,"prepare.run_cpu_time_us":186,"prepare.run_wall_time_us":65261,"replication_time_us":1134269,"thread_start_us":45,"threads_started":1}]]}
I20260808 07:02:38.181458 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8; no voters: 9b54f999704c45bca763910c6647c87a, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:38.188699 20264 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:38.737237 20264 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:38.737382 20264 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:38.737627 20264 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:38.737845 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:38.737892 20133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:02:38.738142 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8; no voters: 9b54f999704c45bca763910c6647c87a, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:39.113797 20263 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:39.114005 20132 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.15.250.129:40075 (request call id 2333) took 1141 ms. Trace:
I20260808 07:02:39.114089 20132 rpcz_store.cc:276] 0808 07:02:37.972036 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:37.972057 (+ 21us) service_pool.cc:227] Handling call
0808 07:02:37.972100 (+ 43us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 07:02:37.972117 (+ 17us) raft_consensus.cc:1520] Early marking committed up to index 9
0808 07:02:37.972210 (+ 93us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 07:02:37.972226 (+ 16us) write_op.cc:270] Start()
0808 07:02:38.181313 (+209087us) write_op.cc:276] Timestamp: P: 1786172556981564 usec, L: 0
0808 07:02:38.181370 (+ 57us) write_op.cc:270] Start()
0808 07:02:39.113488 (+932118us) write_op.cc:276] Timestamp: P: 1786172557632317 usec, L: 0
0808 07:02:39.113650 (+ 162us) log.cc:842] Serialized 708 byte log entry
0808 07:02:39.113748 (+ 98us) raft_consensus.cc:1631] Marking committed up to 9
0808 07:02:39.113749 (+ 1us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 07:02:39.113762 (+ 13us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 07:02:39.113987 (+ 225us) raft_consensus.cc:1671] finished
0808 07:02:39.113987 (+ 0us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 07:02:39.114000 (+ 13us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:38.181391 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:38.181402 (+ 11us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:38.181402 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:38.181403 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:38.181413 (+ 10us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:38.181417 (+ 4us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:38.181418 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:38.181426 (+ 8us) tablet.cc:699] Row locks acquired
0808 07:02:38.181426 (+ 0us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0808 07:02:39.113671 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:39.113686 (+ 15us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:39.113686 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:39.113687 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:39.113698 (+ 11us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:39.113715 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:39.113716 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:39.113725 (+ 9us) tablet.cc:699] Row locks acquired
0808 07:02:39.113726 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":87,"threads_started":1,"wal-append.queue_time_us":155,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":1,"prepare.queue_time_us":46,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":50}],["op",{"num_ops":1,"partition_lock_wait_us":14,"prepare.queue_time_us":131,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":82,"thread_start_us":70,"threads_started":1}]]}
I20260808 07:02:39.199466 20264 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:39.200153 20285 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=11, attempt_no=0}) took 1010 ms. Trace:
I20260808 07:02:39.200248 20285 rpcz_store.cc:276] 0808 07:02:38.190013 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:38.190035 (+ 22us) service_pool.cc:227] Handling call
0808 07:02:39.200143 (+1010108us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:38.210415 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:38.210430 (+ 15us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:38.210430 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:38.210431 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:38.210461 (+ 30us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:38.210463 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:38.210464 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:38.210475 (+ 11us) tablet.cc:699] Row locks acquired
0808 07:02:38.210475 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:38.210483 (+ 8us) write_op.cc:270] Start()
0808 07:02:38.607606 (+397123us) write_op.cc:276] Timestamp: P: 1786172558210446 usec, L: 0
0808 07:02:38.607609 (+ 3us) op_driver.cc:331] REPLICATION: starting
0808 07:02:38.607674 (+ 65us) log.cc:842] Serialized 354 byte log entry
0808 07:02:39.199948 (+592274us) op_driver.cc:444] REPLICATION: finished
0808 07:02:39.200020 (+ 72us) write_op.cc:301] APPLY: starting
0808 07:02:39.200025 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:39.200029 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:39.200033 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:39.200050 (+ 17us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:39.200056 (+ 6us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:39.200057 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:39.200085 (+ 28us) log.cc:842] Serialized 43 byte log entry
0808 07:02:39.200106 (+ 21us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:39.200109 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:39.200110 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":54,"num_ops":1,"prepare.queue_time_us":20298,"prepare.run_cpu_time_us":197,"prepare.run_wall_time_us":397282,"replication_time_us":592335}]]}
I20260808 07:02:39.692764 20264 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:39.692870 20264 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:39.693073 20264 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:39.693264 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:39.693324 20133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:02:39.693675 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8; no voters: 9b54f999704c45bca763910c6647c87a, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:39.693805 20264 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:40.698432 20264 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:40.698566 20264 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:40.698812 20264 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:40.698997 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:40.699057 20133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:02:40.699383 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8; no voters: 9b54f999704c45bca763910c6647c87a, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:40.699407 20296 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=13, attempt_no=0}) took 1498 ms. Trace:
I20260808 07:02:40.699461 20296 rpcz_store.cc:276] 0808 07:02:39.200996 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:39.201043 (+ 47us) service_pool.cc:227] Handling call
0808 07:02:40.699396 (+1498353us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:39.201217 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:39.201494 (+ 277us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:39.201495 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:39.201495 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:39.201506 (+ 11us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:39.201509 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:39.201510 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:39.201518 (+ 8us) tablet.cc:699] Row locks acquired
0808 07:02:39.201518 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:39.201525 (+ 7us) write_op.cc:270] Start()
0808 07:02:39.753623 (+552098us) write_op.cc:276] Timestamp: P: 1786172559201498 usec, L: 0
0808 07:02:39.753625 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:02:39.753676 (+ 51us) log.cc:842] Serialized 354 byte log entry
0808 07:02:40.699121 (+945445us) op_driver.cc:444] REPLICATION: finished
0808 07:02:40.699289 (+ 168us) write_op.cc:301] APPLY: starting
0808 07:02:40.699294 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:40.699298 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:40.699302 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:40.699322 (+ 20us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:40.699327 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:40.699327 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:02:40.699349 (+ 22us) log.cc:842] Serialized 43 byte log entry
0808 07:02:40.699366 (+ 17us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:40.699369 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:40.699369 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":156,"num_ops":1,"prepare.queue_time_us":62,"prepare.run_cpu_time_us":447,"prepare.run_wall_time_us":552517,"raft.queue_time_us":33,"raft.run_cpu_time_us":65,"raft.run_wall_time_us":65,"replication_time_us":945492,"thread_start_us":74,"threads_started":1}]]}
I20260808 07:02:40.943194 20263 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:40.943311 20263 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:40.943413 20132 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.15.250.129:40075 (request call id 2334) took 1828 ms. Trace:
I20260808 07:02:40.943513 20263 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897)
I20260808 07:02:40.943470 20132 rpcz_store.cc:276] 0808 07:02:39.114816 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:39.114850 (+ 34us) service_pool.cc:227] Handling call
0808 07:02:39.114895 (+ 45us) raft_consensus.cc:1457] Updating replica for 3 ops
0808 07:02:39.114913 (+ 18us) raft_consensus.cc:1520] Early marking committed up to index 11
0808 07:02:39.115055 (+ 142us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0808 07:02:39.115100 (+ 45us) write_op.cc:270] Start()
0808 07:02:39.444322 (+329222us) write_op.cc:276] Timestamp: P: 1786172558182155 usec, L: 0
0808 07:02:39.444403 (+ 81us) write_op.cc:270] Start()
0808 07:02:40.305522 (+861119us) write_op.cc:276] Timestamp: P: 1786172558210446 usec, L: 0
0808 07:02:40.305702 (+ 180us) write_op.cc:270] Start()
0808 07:02:40.942830 (+637128us) write_op.cc:276] Timestamp: P: 1786172558738989 usec, L: 0
0808 07:02:40.942999 (+ 169us) log.cc:842] Serialized 1062 byte log entry
0808 07:02:40.943072 (+ 73us) raft_consensus.cc:1631] Marking committed up to 12
0808 07:02:40.943142 (+ 70us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 07:02:40.943157 (+ 15us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 07:02:40.943385 (+ 228us) raft_consensus.cc:1671] finished
0808 07:02:40.943386 (+ 1us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 07:02:40.943404 (+ 18us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:39.444428 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:39.444442 (+ 14us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:39.444442 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:39.444443 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:39.444457 (+ 14us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:39.444461 (+ 4us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:39.444463 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:39.444472 (+ 9us) tablet.cc:699] Row locks acquired
0808 07:02:39.444472 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:40.943079 (+1498607us) op_driver.cc:444] REPLICATION: finished
0808 07:02:40.943185 (+ 106us) write_op.cc:301] APPLY: starting
0808 07:02:40.943188 (+ 3us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:40.943193 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:40.943193 (+ 0us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:40.943207 (+ 14us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:40.943213 (+ 6us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:40.943216 (+ 3us) write_op.cc:312] APPLY: finished
0808 07:02:40.943235 (+ 19us) log.cc:842] Serialized 22 byte log entry
0808 07:02:40.943242 (+ 7us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:40.943245 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:40.943246 (+ 1us) write_op.cc:341] FINISH: Updating metrics
0808 07:02:40.943262 (+ 16us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:40.943262 (+ 0us) write_op.cc:454] Released schema lock
Related trace 'op':
0808 07:02:40.305731 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:40.305749 (+ 18us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:40.305750 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:40.305750 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:40.305765 (+ 15us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:40.305768 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:40.305769 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:40.305782 (+ 13us) tablet.cc:699] Row locks acquired
0808 07:02:40.305782 (+ 0us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0808 07:02:40.943025 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:40.943038 (+ 13us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:40.943039 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:40.943039 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:40.943051 (+ 12us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:40.943054 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:40.943055 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:40.943064 (+ 9us) tablet.cc:699] Row locks acquired
0808 07:02:40.943065 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":64,"threads_started":1,"wal-append.queue_time_us":107,"child_traces":[["op",{"apply.queue_time_us":95,"apply.run_cpu_time_us":88,"apply.run_wall_time_us":89,"num_ops":1,"partition_lock_wait_us":1,"prepare.queue_time_us":46,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":76,"replication_time_us":1827976,"thread_start_us":56,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":1,"prepare.queue_time_us":153,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":82,"thread_start_us":96,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":1,"prepare.queue_time_us":146,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":56,"thread_start_us":80,"threads_started":1}]]}
I20260808 07:02:40.943810 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:40.943862 19992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" is_pre_election: true
I20260808 07:02:40.944128 20062 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b54f999704c45bca763910c6647c87a; no voters: 886bd7a0d91c4ca6947754a55fa13bf8, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:40.944218 20263 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:41.428890 20264 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:41.429684 20306 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=14, attempt_no=0}) took 1735 ms. Trace:
I20260808 07:02:41.429757 20306 rpcz_store.cc:276] 0808 07:02:39.693863 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:39.693889 (+ 26us) service_pool.cc:227] Handling call
0808 07:02:41.429669 (+1735780us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:39.753740 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:39.753756 (+ 16us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:39.753756 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:39.753757 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:39.753770 (+ 13us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:39.753773 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:39.753773 (+ 0us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:39.753785 (+ 12us) tablet.cc:699] Row locks acquired
0808 07:02:39.753785 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:39.753792 (+ 7us) write_op.cc:270] Start()
0808 07:02:39.964912 (+211120us) write_op.cc:276] Timestamp: P: 1786172559753749 usec, L: 0
0808 07:02:39.964914 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:02:39.964992 (+ 78us) log.cc:842] Serialized 354 byte log entry
0808 07:02:41.429445 (+1464453us) op_driver.cc:444] REPLICATION: finished
0808 07:02:41.429555 (+ 110us) write_op.cc:301] APPLY: starting
0808 07:02:41.429559 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:41.429564 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:41.429567 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:41.429581 (+ 14us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:41.429587 (+ 6us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:41.429588 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:41.429607 (+ 19us) log.cc:842] Serialized 22 byte log entry
0808 07:02:41.429640 (+ 33us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:41.429643 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:41.429644 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":98,"num_ops":1,"prepare.queue_time_us":59759,"prepare.run_cpu_time_us":241,"prepare.run_wall_time_us":211318,"replication_time_us":1464526,"thread_start_us":50,"threads_started":1}]]}
I20260808 07:02:42.426203 20264 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:42.426332 20264 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:42.426515 20264 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:42.426786 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:42.426788 20133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:02:42.427078 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8; no voters: 9b54f999704c45bca763910c6647c87a, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:42.427181 20264 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:42.427287 20313 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=15, attempt_no=0}) took 1727 ms. Trace:
I20260808 07:02:42.427338 20313 rpcz_store.cc:276] 0808 07:02:40.700117 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:40.700160 (+ 43us) service_pool.cc:227] Handling call
0808 07:02:42.427269 (+1727109us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:40.700363 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:40.700375 (+ 12us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:40.700376 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:40.700376 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:40.700386 (+ 10us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:40.700388 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:40.700389 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:40.700396 (+ 7us) tablet.cc:699] Row locks acquired
0808 07:02:40.700396 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:40.700402 (+ 6us) write_op.cc:270] Start()
0808 07:02:40.805513 (+105111us) write_op.cc:276] Timestamp: P: 1786172560700388 usec, L: 0
0808 07:02:40.805516 (+ 3us) op_driver.cc:331] REPLICATION: starting
0808 07:02:40.805592 (+ 76us) log.cc:842] Serialized 354 byte log entry
0808 07:02:42.427053 (+1621461us) op_driver.cc:444] REPLICATION: finished
0808 07:02:42.427126 (+ 73us) write_op.cc:301] APPLY: starting
0808 07:02:42.427129 (+ 3us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:42.427133 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:42.427138 (+ 5us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:42.427174 (+ 36us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:42.427178 (+ 4us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:42.427178 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:02:42.427221 (+ 43us) log.cc:842] Serialized 43 byte log entry
0808 07:02:42.427236 (+ 15us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:42.427244 (+ 8us) write_op.cc:454] Released schema lock
0808 07:02:42.427244 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":64,"num_ops":1,"prepare.queue_time_us":125,"prepare.run_cpu_time_us":197,"prepare.run_wall_time_us":105271,"replication_time_us":1621533,"thread_start_us":111,"threads_started":2}]]}
I20260808 07:02:42.880615 20263 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:42.880652 20132 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.15.250.129:40075 (request call id 2335) took 1936 ms. Trace:
I20260808 07:02:42.880728 20263 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:42.881006 20263 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897)
I20260808 07:02:42.881186 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:42.881199 19991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" is_pre_election: true
I20260808 07:02:42.880849 20132 rpcz_store.cc:276] 0808 07:02:40.944354 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:40.944377 (+ 23us) service_pool.cc:227] Handling call
0808 07:02:40.944417 (+ 40us) raft_consensus.cc:1457] Updating replica for 3 ops
0808 07:02:40.944438 (+ 21us) raft_consensus.cc:1520] Early marking committed up to index 14
0808 07:02:40.944532 (+ 94us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0808 07:02:40.944572 (+ 40us) write_op.cc:270] Start()
0808 07:02:41.279676 (+335104us) write_op.cc:276] Timestamp: P: 1786172559201498 usec, L: 0
0808 07:02:41.279744 (+ 68us) write_op.cc:270] Start()
0808 07:02:41.949852 (+670108us) write_op.cc:276] Timestamp: P: 1786172559753749 usec, L: 0
0808 07:02:41.950015 (+ 163us) write_op.cc:270] Start()
0808 07:02:42.880120 (+930105us) write_op.cc:276] Timestamp: P: 1786172560700388 usec, L: 0
0808 07:02:42.880311 (+ 191us) log.cc:842] Serialized 1062 byte log entry
0808 07:02:42.880396 (+ 85us) raft_consensus.cc:1631] Marking committed up to 15
0808 07:02:42.880449 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 07:02:42.880462 (+ 13us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 07:02:42.880639 (+ 177us) raft_consensus.cc:1671] finished
0808 07:02:42.880639 (+ 0us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 07:02:42.880648 (+ 9us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:41.279767 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:41.279779 (+ 12us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:41.279780 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:41.279780 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:41.279792 (+ 12us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:41.279796 (+ 4us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:41.279797 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:41.279806 (+ 9us) tablet.cc:699] Row locks acquired
0808 07:02:41.279806 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:42.880404 (+1600598us) op_driver.cc:444] REPLICATION: finished
0808 07:02:42.880499 (+ 95us) write_op.cc:301] APPLY: starting
0808 07:02:42.880504 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:42.880509 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:42.880510 (+ 1us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:42.880528 (+ 18us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:42.880532 (+ 4us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:42.880537 (+ 5us) write_op.cc:312] APPLY: finished
0808 07:02:42.880554 (+ 17us) log.cc:842] Serialized 43 byte log entry
0808 07:02:42.880558 (+ 4us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:42.880561 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:42.880561 (+ 0us) write_op.cc:341] FINISH: Updating metrics
0808 07:02:42.880679 (+ 118us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:42.880680 (+ 1us) write_op.cc:454] Released schema lock
Related trace 'op':
0808 07:02:41.950051 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:41.950065 (+ 14us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:41.950066 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:41.950066 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:41.950079 (+ 13us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:41.950082 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:41.950082 (+ 0us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:41.950092 (+ 10us) tablet.cc:699] Row locks acquired
0808 07:02:41.950093 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0808 07:02:42.880308 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:42.880322 (+ 14us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:42.880322 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:42.880323 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:42.880334 (+ 11us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:42.880337 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:42.880339 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:42.880366 (+ 27us) tablet.cc:699] Row locks acquired
0808 07:02:42.880367 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":75,"threads_started":1,"wal-append.queue_time_us":138,"child_traces":[["op",{"apply.queue_time_us":85,"apply.run_cpu_time_us":97,"apply.run_wall_time_us":194,"num_ops":1,"partition_lock_wait_us":1,"prepare.queue_time_us":40,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":77,"replication_time_us":1935828,"thread_start_us":38,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":1,"prepare.queue_time_us":152,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":66,"thread_start_us":95,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":1,"prepare.queue_time_us":143,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":75,"thread_start_us":67,"threads_started":1}]]}
I20260808 07:02:42.881743 20062 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b54f999704c45bca763910c6647c87a; no voters: 886bd7a0d91c4ca6947754a55fa13bf8, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:42.881837 20263 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:43.454208 20263 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:43.454327 20263 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:43.454514 20263 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897)
I20260808 07:02:43.454756 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:43.454807 19992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" is_pre_election: true
I20260808 07:02:43.455021 20062 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b54f999704c45bca763910c6647c87a; no voters: 886bd7a0d91c4ca6947754a55fa13bf8, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:43.455148 20263 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:43.455271 20322 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=16, attempt_no=0}) took 2024 ms. Trace:
I20260808 07:02:43.455328 20322 rpcz_store.cc:276] 0808 07:02:41.430508 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:41.430568 (+ 60us) service_pool.cc:227] Handling call
0808 07:02:43.455260 (+2024692us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:41.430820 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:41.430831 (+ 11us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:41.430831 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:41.430832 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:41.430843 (+ 11us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:41.430845 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:41.430846 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:42.427260 (+996414us) lock_manager.cc:599] Waited 996363us for lock on <redacted> (key hash 17251042410248443849)
0808 07:02:42.427263 (+ 3us) tablet.cc:699] Row locks acquired
0808 07:02:42.427264 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:02:42.427274 (+ 10us) write_op.cc:270] Start()
0808 07:02:42.968375 (+541101us) write_op.cc:276] Timestamp: P: 1786172562427244 usec, L: 0
0808 07:02:42.968377 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:02:42.968423 (+ 46us) log.cc:842] Serialized 354 byte log entry
0808 07:02:43.454758 (+486335us) op_driver.cc:444] REPLICATION: finished
0808 07:02:43.455074 (+ 316us) write_op.cc:301] APPLY: starting
0808 07:02:43.455096 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:43.455101 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:43.455101 (+ 0us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:43.455116 (+ 15us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:43.455120 (+ 4us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:43.455121 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:43.455144 (+ 23us) log.cc:842] Serialized 43 byte log entry
0808 07:02:43.455230 (+ 86us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:43.455233 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:43.455234 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":173,"num_ops":1,"prepare.queue_time_us":87,"prepare.run_cpu_time_us":239,"prepare.run_wall_time_us":1537652,"raft.queue_time_us":21,"raft.run_cpu_time_us":48,"raft.run_wall_time_us":48,"replication_time_us":486376,"row_lock_wait_count":1,"row_lock_wait_us":996363,"thread_start_us":217,"threads_started":2}]]}
I20260808 07:02:43.485251 20320 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:43.485375 20320 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:43.485608 20320 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:43.485800 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:43.485818 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:02:43.486055 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8; no voters: 9b54f999704c45bca763910c6647c87a, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:43.486189 20320 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:44.331646 20320 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:44.331785 20320 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:44.332000 20320 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:44.332242 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:44.332490 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:02:44.332717 20331 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=17, attempt_no=0}) took 1904 ms. Trace:
I20260808 07:02:44.332800 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8; no voters: 9b54f999704c45bca763910c6647c87a, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:44.332787 20331 rpcz_store.cc:276] 0808 07:02:42.427935 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:42.427977 (+ 42us) service_pool.cc:227] Handling call
0808 07:02:44.332700 (+1904723us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:42.968478 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:42.968494 (+ 16us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:42.968494 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:42.968495 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:42.968508 (+ 13us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:42.968511 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:42.968512 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:42.968616 (+ 104us) tablet.cc:699] Row locks acquired
0808 07:02:42.968616 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:42.968626 (+ 10us) write_op.cc:270] Start()
0808 07:02:43.808751 (+840125us) write_op.cc:276] Timestamp: P: 1786172562968585 usec, L: 0
0808 07:02:43.808753 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:02:43.808812 (+ 59us) log.cc:842] Serialized 354 byte log entry
0808 07:02:44.332415 (+523603us) op_driver.cc:444] REPLICATION: finished
0808 07:02:44.332549 (+ 134us) write_op.cc:301] APPLY: starting
0808 07:02:44.332553 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:44.332558 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:44.332561 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:44.332578 (+ 17us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:44.332584 (+ 6us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:44.332594 (+ 10us) write_op.cc:312] APPLY: finished
0808 07:02:44.332648 (+ 54us) log.cc:842] Serialized 43 byte log entry
0808 07:02:44.332669 (+ 21us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:44.332672 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:44.332672 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":125,"num_ops":1,"prepare.queue_time_us":540383,"prepare.run_cpu_time_us":214,"prepare.run_wall_time_us":840383,"replication_time_us":523658,"spinlock_wait_cycles":768,"thread_start_us":54,"threads_started":1}]]}
I20260808 07:02:44.443466 20263 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:44.443570 20263 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:44.443791 20263 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897)
I20260808 07:02:44.444017 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:44.444046 19992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" is_pre_election: true
I20260808 07:02:44.444309 20062 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b54f999704c45bca763910c6647c87a; no voters: 886bd7a0d91c4ca6947754a55fa13bf8, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:44.689697 20263 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:44.690510 20331 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=18, attempt_no=0}) took 1234 ms. Trace:
I20260808 07:02:44.690593 20331 rpcz_store.cc:276] 0808 07:02:43.455940 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:43.456004 (+ 64us) service_pool.cc:227] Handling call
0808 07:02:44.690497 (+1234493us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:43.808869 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:43.808885 (+ 16us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:43.808885 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:43.808886 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:43.808900 (+ 14us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:43.808902 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:43.808904 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:43.808915 (+ 11us) tablet.cc:699] Row locks acquired
0808 07:02:43.808915 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:43.808923 (+ 8us) write_op.cc:270] Start()
0808 07:02:44.232033 (+423110us) write_op.cc:276] Timestamp: P: 1786172563808907 usec, L: 0
0808 07:02:44.232035 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:02:44.232104 (+ 69us) log.cc:842] Serialized 354 byte log entry
0808 07:02:44.690256 (+458152us) op_driver.cc:444] REPLICATION: finished
0808 07:02:44.690365 (+ 109us) write_op.cc:301] APPLY: starting
0808 07:02:44.690369 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:44.690373 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:44.690376 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:44.690399 (+ 23us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:44.690403 (+ 4us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:44.690422 (+ 19us) write_op.cc:312] APPLY: finished
0808 07:02:44.690447 (+ 25us) log.cc:842] Serialized 22 byte log entry
0808 07:02:44.690469 (+ 22us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:44.690472 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:44.690473 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":84,"num_ops":1,"prepare.queue_time_us":352777,"prepare.run_cpu_time_us":202,"prepare.run_wall_time_us":423276,"replication_time_us":458216}]]}
I20260808 07:02:44.817160 20320 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:44.879527 20263 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:44.879645 20263 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:44.879954 20263 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897)
I20260808 07:02:44.880242 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:44.880532 19992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" is_pre_election: true
I20260808 07:02:44.880813 20062 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b54f999704c45bca763910c6647c87a; no voters: 886bd7a0d91c4ca6947754a55fa13bf8, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:44.880930 20263 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:45.200500 20320 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:45.200630 20320 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:45.200822 20320 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:45.200978 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:45.200997 20133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:02:45.201273 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8; no voters: 9b54f999704c45bca763910c6647c87a, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:45.201368 20320 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:45.983089 20337 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:45.983215 20337 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:45.983469 20337 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:45.983690 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:45.983727 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:02:45.983932 20339 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=20, attempt_no=0}) took 1292 ms. Trace:
I20260808 07:02:45.984010 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8; no voters: 9b54f999704c45bca763910c6647c87a, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:45.984112 20337 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:45.984002 20339 rpcz_store.cc:276] 0808 07:02:44.691111 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:44.691157 (+ 46us) service_pool.cc:227] Handling call
0808 07:02:45.983915 (+1292758us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:44.836895 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:44.836910 (+ 15us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:44.836911 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:44.836911 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:44.836925 (+ 14us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:44.836927 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:44.836929 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:44.836939 (+ 10us) tablet.cc:699] Row locks acquired
0808 07:02:44.836939 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:44.836947 (+ 8us) write_op.cc:270] Start()
0808 07:02:45.806046 (+969099us) write_op.cc:276] Timestamp: P: 1786172564836937 usec, L: 0
0808 07:02:45.806048 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:02:45.806108 (+ 60us) log.cc:842] Serialized 354 byte log entry
0808 07:02:45.983558 (+177450us) op_driver.cc:444] REPLICATION: finished
0808 07:02:45.983725 (+ 167us) write_op.cc:301] APPLY: starting
0808 07:02:45.983730 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:45.983734 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:45.983737 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:45.983750 (+ 13us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:45.983755 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:45.983756 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:45.983773 (+ 17us) log.cc:842] Serialized 22 byte log entry
0808 07:02:45.983816 (+ 43us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:45.983893 (+ 77us) write_op.cc:454] Released schema lock
0808 07:02:45.983894 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":155,"num_ops":1,"prepare.queue_time_us":145474,"prepare.run_cpu_time_us":198,"prepare.run_wall_time_us":969249,"replication_time_us":177506,"thread_start_us":65,"threads_started":1}]]}
I20260808 07:02:46.253101 20337 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:46.253216 20337 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:46.253414 20337 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:46.253619 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:46.253655 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:02:46.253778 20339 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=21, attempt_no=0}) took 1373 ms. Trace:
I20260808 07:02:46.253912 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8; no voters: 9b54f999704c45bca763910c6647c87a, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:46.253849 20339 rpcz_store.cc:276] 0808 07:02:44.880539 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:44.880564 (+ 25us) service_pool.cc:227] Handling call
0808 07:02:46.253772 (+1373208us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:45.806157 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:45.806172 (+ 15us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:45.806173 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:45.806173 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:45.806186 (+ 13us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:45.806189 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:45.806190 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:45.806200 (+ 10us) tablet.cc:699] Row locks acquired
0808 07:02:45.806200 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:45.806208 (+ 8us) write_op.cc:270] Start()
0808 07:02:46.197418 (+391210us) write_op.cc:276] Timestamp: P: 1786172565806193 usec, L: 0
0808 07:02:46.197420 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:02:46.197485 (+ 65us) log.cc:842] Serialized 354 byte log entry
0808 07:02:46.253629 (+ 56144us) op_driver.cc:444] REPLICATION: finished
0808 07:02:46.253667 (+ 38us) write_op.cc:301] APPLY: starting
0808 07:02:46.253671 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:46.253676 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:46.253679 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:46.253699 (+ 20us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:46.253704 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:46.253705 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:46.253728 (+ 23us) log.cc:842] Serialized 43 byte log entry
0808 07:02:46.253745 (+ 17us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:46.253748 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:46.253748 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":20,"num_ops":1,"prepare.queue_time_us":925399,"prepare.run_cpu_time_us":299,"prepare.run_wall_time_us":391365,"replication_time_us":56205}]]}
I20260808 07:02:46.254097 20337 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:46.562207 20336 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:46.562340 20336 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:46.562572 20336 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897)
I20260808 07:02:46.562832 19991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" is_pre_election: true
I20260808 07:02:46.562810 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:46.563195 20062 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b54f999704c45bca763910c6647c87a; no voters: 886bd7a0d91c4ca6947754a55fa13bf8, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:46.865700 20345 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:46.865811 20345 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:46.866156 20345 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:46.866400 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:46.866506 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:02:46.866732 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8; no voters: 9b54f999704c45bca763910c6647c87a, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:47.325877 20336 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:47.834858 20345 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:47.835749 20354 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=23, attempt_no=0}) took 1581 ms. Trace:
I20260808 07:02:47.835834 20354 rpcz_store.cc:276] 0808 07:02:46.254457 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:46.254537 (+ 80us) service_pool.cc:227] Handling call
0808 07:02:47.835718 (+1581181us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:46.769824 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:46.769848 (+ 24us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:46.769849 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:46.769849 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:46.769867 (+ 18us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:46.769869 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:46.769871 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:46.769881 (+ 10us) tablet.cc:699] Row locks acquired
0808 07:02:46.769881 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:46.769891 (+ 10us) write_op.cc:270] Start()
0808 07:02:46.803029 (+ 33138us) write_op.cc:276] Timestamp: P: 1786172566769831 usec, L: 0
0808 07:02:46.803032 (+ 3us) op_driver.cc:331] REPLICATION: starting
0808 07:02:46.803101 (+ 69us) log.cc:842] Serialized 354 byte log entry
0808 07:02:47.835476 (+1032375us) op_driver.cc:444] REPLICATION: finished
0808 07:02:47.835597 (+ 121us) write_op.cc:301] APPLY: starting
0808 07:02:47.835601 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:47.835606 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:47.835609 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:47.835628 (+ 19us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:47.835634 (+ 6us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:47.835634 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:02:47.835660 (+ 26us) log.cc:842] Serialized 43 byte log entry
0808 07:02:47.835681 (+ 21us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:47.835684 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:47.835685 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":106,"num_ops":1,"prepare.queue_time_us":515156,"prepare.run_cpu_time_us":217,"prepare.run_wall_time_us":33300,"replication_time_us":1032440,"thread_start_us":48,"threads_started":1}]]}
I20260808 07:02:47.877651 20336 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:47.877756 20336 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:47.877939 20336 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897)
I20260808 07:02:47.878177 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:47.878288 19992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" is_pre_election: true
I20260808 07:02:47.878556 20062 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b54f999704c45bca763910c6647c87a; no voters: 886bd7a0d91c4ca6947754a55fa13bf8, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:48.003304 20345 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:48.003580 20345 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:48.003809 20345 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:48.004141 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:02:48.004171 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:48.004226 20354 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=24, attempt_no=0}) took 1137 ms. Trace:
I20260808 07:02:48.004318 20354 rpcz_store.cc:276] 0808 07:02:46.867113 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:46.867159 (+ 46us) service_pool.cc:227] Handling call
0808 07:02:48.004217 (+1137058us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:46.867351 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:46.867360 (+ 9us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:46.867361 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:46.867361 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:46.867369 (+ 8us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:46.867371 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:46.867372 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:46.867380 (+ 8us) tablet.cc:699] Row locks acquired
0808 07:02:46.867380 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:46.867384 (+ 4us) write_op.cc:270] Start()
0808 07:02:47.369501 (+502117us) write_op.cc:276] Timestamp: P: 1786172566867349 usec, L: 0
0808 07:02:47.369503 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:02:47.369570 (+ 67us) log.cc:842] Serialized 354 byte log entry
0808 07:02:48.003801 (+634231us) op_driver.cc:444] REPLICATION: finished
0808 07:02:48.003881 (+ 80us) write_op.cc:301] APPLY: starting
0808 07:02:48.003886 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:48.003891 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:48.003895 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:48.003924 (+ 29us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:48.004125 (+ 201us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:48.004126 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:48.004144 (+ 18us) log.cc:842] Serialized 22 byte log entry
0808 07:02:48.004193 (+ 49us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:48.004197 (+ 4us) write_op.cc:454] Released schema lock
0808 07:02:48.004198 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":69,"num_ops":1,"prepare.queue_time_us":40,"prepare.run_cpu_time_us":181,"prepare.run_wall_time_us":502265,"replication_time_us":634293}]]}
I20260808 07:02:48.004490 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8; no voters: 9b54f999704c45bca763910c6647c87a, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:48.004591 20345 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:48.870661 20336 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:49.134761 20336 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:49.134881 20336 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:49.135084 20336 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897)
I20260808 07:02:49.135298 19992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" is_pre_election: true
I20260808 07:02:49.135277 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:49.135643 20062 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b54f999704c45bca763910c6647c87a; no voters: 886bd7a0d91c4ca6947754a55fa13bf8, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:49.135740 20336 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:49.135701 20361 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=25, attempt_no=0}) took 1299 ms. Trace:
I20260808 07:02:49.135878 20361 rpcz_store.cc:276] 0808 07:02:47.836642 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:47.836676 (+ 34us) service_pool.cc:227] Handling call
0808 07:02:49.135692 (+1299016us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:47.836841 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:47.836853 (+ 12us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:47.836854 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:47.836854 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:47.836864 (+ 10us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:47.836866 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:47.836867 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:47.836874 (+ 7us) tablet.cc:699] Row locks acquired
0808 07:02:47.836875 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:02:47.836878 (+ 3us) write_op.cc:270] Start()
0808 07:02:48.612975 (+776097us) write_op.cc:276] Timestamp: P: 1786172567836866 usec, L: 0
0808 07:02:48.612976 (+ 1us) op_driver.cc:331] REPLICATION: starting
0808 07:02:48.613023 (+ 47us) log.cc:842] Serialized 354 byte log entry
0808 07:02:49.135451 (+522428us) op_driver.cc:444] REPLICATION: finished
0808 07:02:49.135570 (+ 119us) write_op.cc:301] APPLY: starting
0808 07:02:49.135574 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:49.135579 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:49.135581 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:49.135604 (+ 23us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:49.135608 (+ 4us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:49.135609 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:49.135627 (+ 18us) log.cc:842] Serialized 43 byte log entry
0808 07:02:49.135662 (+ 35us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:49.135665 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:49.135666 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":107,"num_ops":1,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":161,"prepare.run_wall_time_us":776228,"raft.queue_time_us":29,"raft.run_cpu_time_us":32,"raft.run_wall_time_us":32,"replication_time_us":522471,"thread_start_us":32,"threads_started":1}]]}
I20260808 07:02:49.535796 20357 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:49.535907 20357 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:49.536125 20357 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:49.536375 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:49.536413 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:02:49.536674 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8; no voters: 9b54f999704c45bca763910c6647c87a, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:49.823114 20336 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:49.823241 20336 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:49.823441 20336 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897)
I20260808 07:02:49.823663 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:49.823721 19992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" is_pre_election: true
I20260808 07:02:49.824064 20062 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b54f999704c45bca763910c6647c87a; no voters: 886bd7a0d91c4ca6947754a55fa13bf8, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:49.824506 20367 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=26, attempt_no=0}) took 1819 ms. Trace:
I20260808 07:02:49.824568 20367 rpcz_store.cc:276] 0808 07:02:48.004908 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:48.004931 (+ 23us) service_pool.cc:227] Handling call
0808 07:02:49.824012 (+1819081us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:48.613076 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:48.613093 (+ 17us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:48.613094 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:48.613095 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:48.613113 (+ 18us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:48.613116 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:48.613117 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:48.613128 (+ 11us) tablet.cc:699] Row locks acquired
0808 07:02:48.613128 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:48.613137 (+ 9us) write_op.cc:270] Start()
0808 07:02:49.278238 (+665101us) write_op.cc:276] Timestamp: P: 1786172568613097 usec, L: 0
0808 07:02:49.278240 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:02:49.278291 (+ 51us) log.cc:842] Serialized 354 byte log entry
0808 07:02:49.823746 (+545455us) op_driver.cc:444] REPLICATION: finished
0808 07:02:49.823882 (+ 136us) write_op.cc:301] APPLY: starting
0808 07:02:49.823886 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:49.823890 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:49.823893 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:49.823907 (+ 14us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:49.823912 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:49.823913 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:49.823958 (+ 45us) log.cc:842] Serialized 43 byte log entry
0808 07:02:49.823971 (+ 13us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:49.823974 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:49.823975 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":126,"num_ops":1,"prepare.queue_time_us":608041,"prepare.run_cpu_time_us":198,"prepare.run_wall_time_us":665266,"raft.queue_time_us":41,"raft.run_cpu_time_us":71,"raft.run_wall_time_us":71,"replication_time_us":545501,"thread_start_us":56,"threads_started":1}]]}
I20260808 07:02:49.978514 20336 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:50.857448 20357 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:50.857635 19991 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.15.250.129:48641 (request call id 4176) took 1321 ms. Trace:
I20260808 07:02:50.857728 19991 rpcz_store.cc:276] 0808 07:02:49.536545 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:49.536568 (+ 23us) service_pool.cc:227] Handling call
0808 07:02:49.536608 (+ 40us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 07:02:49.536626 (+ 18us) raft_consensus.cc:1520] Early marking committed up to index 27
0808 07:02:49.536735 (+ 109us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 07:02:49.536780 (+ 45us) write_op.cc:270] Start()
0808 07:02:50.028922 (+492142us) write_op.cc:276] Timestamp: P: 1786172568613097 usec, L: 0
0808 07:02:50.029020 (+ 98us) write_op.cc:270] Start()
0808 07:02:50.857122 (+828102us) write_op.cc:276] Timestamp: P: 1786172569278357 usec, L: 0
0808 07:02:50.857285 (+ 163us) log.cc:842] Serialized 708 byte log entry
0808 07:02:50.857358 (+ 73us) raft_consensus.cc:1631] Marking committed up to 27
0808 07:02:50.857360 (+ 2us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 07:02:50.857375 (+ 15us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 07:02:50.857611 (+ 236us) raft_consensus.cc:1671] finished
0808 07:02:50.857612 (+ 1us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 07:02:50.857625 (+ 13us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:50.029037 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:50.029055 (+ 18us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:50.029056 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:50.029056 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:50.029070 (+ 14us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:50.029075 (+ 5us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:50.029077 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:50.029087 (+ 10us) tablet.cc:699] Row locks acquired
0808 07:02:50.029087 (+ 0us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0808 07:02:50.857290 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:50.857305 (+ 15us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:50.857306 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:50.857306 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:50.857319 (+ 13us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:50.857344 (+ 25us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:50.857345 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:50.857354 (+ 9us) tablet.cc:699] Row locks acquired
0808 07:02:50.857354 (+ 0us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":63,"threads_started":1,"wal-append.queue_time_us":86,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":1,"prepare.queue_time_us":45,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":91}],["op",{"num_ops":1,"partition_lock_wait_us":23,"prepare.queue_time_us":118,"prepare.run_cpu_time_us":135,"prepare.run_wall_time_us":135,"thread_start_us":65,"threads_started":1}]]}
I20260808 07:02:50.900558 20371 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:50.900671 20371 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:50.900929 20371 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897)
I20260808 07:02:50.901106 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:50.901129 19992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" is_pre_election: true
I20260808 07:02:50.901353 20379 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=28, attempt_no=0}) took 1076 ms. Trace:
I20260808 07:02:50.901440 20062 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b54f999704c45bca763910c6647c87a; no voters: 886bd7a0d91c4ca6947754a55fa13bf8, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:50.901521 20371 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:50.901439 20379 rpcz_store.cc:276] 0808 07:02:49.825142 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:49.825220 (+ 78us) service_pool.cc:227] Handling call
0808 07:02:50.901341 (+1076121us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:49.825444 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:49.825455 (+ 11us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:49.825455 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:49.825455 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:49.825464 (+ 9us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:49.825466 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:49.825466 (+ 0us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:49.825473 (+ 7us) tablet.cc:699] Row locks acquired
0808 07:02:49.825473 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:49.825479 (+ 6us) write_op.cc:270] Start()
0808 07:02:50.703580 (+878101us) write_op.cc:276] Timestamp: P: 1786172569825423 usec, L: 0
0808 07:02:50.703581 (+ 1us) op_driver.cc:331] REPLICATION: starting
0808 07:02:50.703631 (+ 50us) log.cc:842] Serialized 354 byte log entry
0808 07:02:50.901142 (+197511us) op_driver.cc:444] REPLICATION: finished
0808 07:02:50.901214 (+ 72us) write_op.cc:301] APPLY: starting
0808 07:02:50.901218 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:50.901222 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:50.901225 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:50.901242 (+ 17us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:50.901246 (+ 4us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:50.901247 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:50.901270 (+ 23us) log.cc:842] Serialized 43 byte log entry
0808 07:02:50.901311 (+ 41us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:50.901314 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:50.901315 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":60,"num_ops":1,"prepare.queue_time_us":108,"prepare.run_cpu_time_us":150,"prepare.run_wall_time_us":878221,"raft.queue_time_us":40,"raft.run_cpu_time_us":45,"raft.run_wall_time_us":46,"replication_time_us":197556,"thread_start_us":96,"threads_started":2}]]}
I20260808 07:02:51.805105 20357 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:51.805230 20357 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:51.805469 20357 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:51.805701 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:51.805699 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:02:51.805991 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8; no voters: 9b54f999704c45bca763910c6647c87a, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:52.078683 20371 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:52.078856 20371 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:52.079042 20371 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897)
I20260808 07:02:52.079226 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:52.079248 19992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" is_pre_election: true
I20260808 07:02:52.079578 20062 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b54f999704c45bca763910c6647c87a; no voters: 886bd7a0d91c4ca6947754a55fa13bf8, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:52.079640 20387 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=29, attempt_no=0}) took 2099 ms. Trace:
I20260808 07:02:52.079731 20387 rpcz_store.cc:276] 0808 07:02:49.979818 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:49.979842 (+ 24us) service_pool.cc:227] Handling call
0808 07:02:52.079623 (+2099781us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:50.703670 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:50.703683 (+ 13us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:50.703684 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:50.703684 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:50.703695 (+ 11us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:50.703697 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:50.703698 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:50.703708 (+ 10us) tablet.cc:699] Row locks acquired
0808 07:02:50.703708 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:50.703715 (+ 7us) write_op.cc:270] Start()
0808 07:02:51.276839 (+573124us) write_op.cc:276] Timestamp: P: 1786172570703676 usec, L: 0
0808 07:02:51.276840 (+ 1us) op_driver.cc:331] REPLICATION: starting
0808 07:02:51.276902 (+ 62us) log.cc:842] Serialized 354 byte log entry
0808 07:02:52.079329 (+802427us) op_driver.cc:444] REPLICATION: finished
0808 07:02:52.079441 (+ 112us) write_op.cc:301] APPLY: starting
0808 07:02:52.079444 (+ 3us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:52.079448 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:52.079451 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:52.079463 (+ 12us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:52.079468 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:52.079469 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:52.079513 (+ 44us) log.cc:842] Serialized 43 byte log entry
0808 07:02:52.079594 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:52.079598 (+ 4us) write_op.cc:454] Released schema lock
0808 07:02:52.079598 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":104,"num_ops":1,"prepare.queue_time_us":723747,"prepare.run_cpu_time_us":218,"prepare.run_wall_time_us":573286,"raft.queue_time_us":29,"raft.run_cpu_time_us":58,"raft.run_wall_time_us":57,"replication_time_us":802484,"thread_start_us":33,"threads_started":1}]]}
I20260808 07:02:52.397557 20357 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:52.478005 20371 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:52.478740 20387 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=30, attempt_no=0}) took 1576 ms. Trace:
I20260808 07:02:52.478813 20387 rpcz_store.cc:276] 0808 07:02:50.902055 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:50.902095 (+ 40us) service_pool.cc:227] Handling call
0808 07:02:52.478729 (+1576634us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:51.276966 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:51.276983 (+ 17us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:51.276983 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:51.276984 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:51.276996 (+ 12us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:51.276999 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:51.277001 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:51.277012 (+ 11us) tablet.cc:699] Row locks acquired
0808 07:02:51.277012 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:51.277021 (+ 9us) write_op.cc:270] Start()
0808 07:02:51.909139 (+632118us) write_op.cc:276] Timestamp: P: 1786172571276969 usec, L: 0
0808 07:02:51.909142 (+ 3us) op_driver.cc:331] REPLICATION: starting
0808 07:02:51.909216 (+ 74us) log.cc:842] Serialized 354 byte log entry
0808 07:02:52.478559 (+569343us) op_driver.cc:444] REPLICATION: finished
0808 07:02:52.478595 (+ 36us) write_op.cc:301] APPLY: starting
0808 07:02:52.478598 (+ 3us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:52.478603 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:52.478606 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:52.478626 (+ 20us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:52.478632 (+ 6us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:52.478632 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:02:52.478656 (+ 24us) log.cc:842] Serialized 43 byte log entry
0808 07:02:52.478673 (+ 17us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:52.478677 (+ 4us) write_op.cc:454] Released schema lock
0808 07:02:52.478677 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":25,"num_ops":1,"prepare.queue_time_us":374749,"prepare.run_cpu_time_us":223,"prepare.run_wall_time_us":632304,"replication_time_us":569412}]]}
I20260808 07:02:53.186321 20371 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:53.186439 20371 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:53.186605 20371 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897)
I20260808 07:02:53.186847 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:53.186932 19992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" is_pre_election: true
I20260808 07:02:53.187188 20062 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b54f999704c45bca763910c6647c87a; no voters: 886bd7a0d91c4ca6947754a55fa13bf8, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:53.187299 20395 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=31, attempt_no=0}) took 1106 ms. Trace:
I20260808 07:02:53.187309 20371 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:53.187394 20395 rpcz_store.cc:276] 0808 07:02:52.080410 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:52.080436 (+ 26us) service_pool.cc:227] Handling call
0808 07:02:53.187289 (+1106853us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:52.080585 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:52.080596 (+ 11us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:52.080596 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:52.080597 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:52.080605 (+ 8us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:52.080606 (+ 1us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:52.080607 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:52.080614 (+ 7us) tablet.cc:699] Row locks acquired
0808 07:02:52.080615 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:02:52.080619 (+ 4us) write_op.cc:270] Start()
0808 07:02:52.121733 (+ 41114us) write_op.cc:276] Timestamp: P: 1786172572080587 usec, L: 0
0808 07:02:52.121736 (+ 3us) op_driver.cc:331] REPLICATION: starting
0808 07:02:52.121801 (+ 65us) log.cc:842] Serialized 354 byte log entry
0808 07:02:53.187009 (+1065208us) op_driver.cc:444] REPLICATION: finished
0808 07:02:53.187158 (+ 149us) write_op.cc:301] APPLY: starting
0808 07:02:53.187169 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:53.187173 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:53.187176 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:53.187190 (+ 14us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:53.187195 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:53.187196 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:53.187219 (+ 23us) log.cc:842] Serialized 43 byte log entry
0808 07:02:53.187254 (+ 35us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:53.187258 (+ 4us) write_op.cc:454] Released schema lock
0808 07:02:53.187259 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":137,"num_ops":1,"prepare.queue_time_us":45,"prepare.run_cpu_time_us":206,"prepare.run_wall_time_us":41281,"replication_time_us":1065269,"thread_start_us":55,"threads_started":1}]]}
I20260808 07:02:53.366997 20357 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:53.367120 20357 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:53.367323 20357 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:53.367512 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:53.367522 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:02:53.367833 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8; no voters: 9b54f999704c45bca763910c6647c87a, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:53.488240 20357 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:53.488981 20395 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=32, attempt_no=0}) took 1009 ms. Trace:
I20260808 07:02:53.489060 20395 rpcz_store.cc:276] 0808 07:02:52.479427 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:52.479479 (+ 52us) service_pool.cc:227] Handling call
0808 07:02:53.488970 (+1009491us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:52.479603 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:52.479617 (+ 14us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:52.479617 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:52.479618 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:52.479627 (+ 9us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:52.479629 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:52.479629 (+ 0us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:52.479636 (+ 7us) tablet.cc:699] Row locks acquired
0808 07:02:52.479636 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:02:52.479640 (+ 4us) write_op.cc:270] Start()
0808 07:02:53.225785 (+746145us) write_op.cc:276] Timestamp: P: 1786172572479606 usec, L: 0
0808 07:02:53.225786 (+ 1us) op_driver.cc:331] REPLICATION: starting
0808 07:02:53.225838 (+ 52us) log.cc:842] Serialized 354 byte log entry
0808 07:02:53.488758 (+262920us) op_driver.cc:444] REPLICATION: finished
0808 07:02:53.488810 (+ 52us) write_op.cc:301] APPLY: starting
0808 07:02:53.488814 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:53.488820 (+ 6us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:53.488824 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:53.488842 (+ 18us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:53.488846 (+ 4us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:53.488847 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:53.488907 (+ 60us) log.cc:842] Serialized 43 byte log entry
0808 07:02:53.488939 (+ 32us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:53.488942 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:53.488942 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":39,"num_ops":1,"prepare.queue_time_us":39,"prepare.run_cpu_time_us":207,"prepare.run_wall_time_us":746296,"raft.queue_time_us":46,"raft.run_cpu_time_us":39,"raft.run_wall_time_us":39,"replication_time_us":262967}]]}
I20260808 07:02:53.489424 20371 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:53.489504 20371 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:53.489720 20371 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897)
I20260808 07:02:53.489959 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:53.490012 19991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" is_pre_election: true
I20260808 07:02:53.490208 20062 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b54f999704c45bca763910c6647c87a; no voters: 886bd7a0d91c4ca6947754a55fa13bf8, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:53.490341 20371 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:54.554100 20357 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:54.554234 20357 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:54.554461 20357 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:54.554719 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:54.554790 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:02:54.555099 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8; no voters: 9b54f999704c45bca763910c6647c87a, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:54.555302 20404 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=33, attempt_no=0}) took 1367 ms. Trace:
I20260808 07:02:54.555364 20404 rpcz_store.cc:276] 0808 07:02:53.188059 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:53.188089 (+ 30us) service_pool.cc:227] Handling call
0808 07:02:54.555294 (+1367205us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:53.225908 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:53.225925 (+ 17us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:53.225926 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:53.225926 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:53.225939 (+ 13us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:53.225942 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:53.225944 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:53.225956 (+ 12us) tablet.cc:699] Row locks acquired
0808 07:02:53.225957 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:02:53.225965 (+ 8us) write_op.cc:270] Start()
0808 07:02:53.909086 (+683121us) write_op.cc:276] Timestamp: P: 1786172573225925 usec, L: 0
0808 07:02:53.909088 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:02:53.909177 (+ 89us) log.cc:842] Serialized 354 byte log entry
0808 07:02:54.554968 (+645791us) op_driver.cc:444] REPLICATION: finished
0808 07:02:54.555075 (+ 107us) write_op.cc:301] APPLY: starting
0808 07:02:54.555079 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:54.555083 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:54.555086 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:54.555107 (+ 21us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:54.555112 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:54.555112 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:02:54.555154 (+ 42us) log.cc:842] Serialized 43 byte log entry
0808 07:02:54.555263 (+ 109us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:54.555266 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:54.555267 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":97,"num_ops":1,"prepare.queue_time_us":37663,"prepare.run_cpu_time_us":275,"prepare.run_wall_time_us":683363,"raft.queue_time_us":122,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":93,"replication_time_us":645875,"spinlock_wait_cycles":8448,"thread_start_us":63,"threads_started":1}]]}
I20260808 07:02:54.592545 20357 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:54.593429 20404 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=34, attempt_no=0}) took 1103 ms. Trace:
I20260808 07:02:54.593531 20404 rpcz_store.cc:276] 0808 07:02:53.489686 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:53.489729 (+ 43us) service_pool.cc:227] Handling call
0808 07:02:54.593419 (+1103690us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:53.909281 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:53.909296 (+ 15us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:53.909297 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:53.909297 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:53.909309 (+ 12us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:53.909312 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:53.909313 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:53.909324 (+ 11us) tablet.cc:699] Row locks acquired
0808 07:02:53.909325 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:02:53.909333 (+ 8us) write_op.cc:270] Start()
0808 07:02:54.325443 (+416110us) write_op.cc:276] Timestamp: P: 1786172573909267 usec, L: 0
0808 07:02:54.325446 (+ 3us) op_driver.cc:331] REPLICATION: starting
0808 07:02:54.325509 (+ 63us) log.cc:842] Serialized 354 byte log entry
0808 07:02:54.593186 (+267677us) op_driver.cc:444] REPLICATION: finished
0808 07:02:54.593243 (+ 57us) write_op.cc:301] APPLY: starting
0808 07:02:54.593247 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:54.593253 (+ 6us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:54.593274 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:54.593289 (+ 15us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:54.593295 (+ 6us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:54.593295 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:02:54.593316 (+ 21us) log.cc:842] Serialized 22 byte log entry
0808 07:02:54.593393 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:54.593397 (+ 4us) write_op.cc:454] Released schema lock
0808 07:02:54.593398 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":41,"num_ops":1,"prepare.queue_time_us":419404,"prepare.run_cpu_time_us":183,"prepare.run_wall_time_us":416253,"replication_time_us":267735}]]}
I20260808 07:02:54.825060 20371 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:54.825170 20371 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:54.825356 20371 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897)
I20260808 07:02:54.825506 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:54.825495 19991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" is_pre_election: true
I20260808 07:02:54.825776 20062 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b54f999704c45bca763910c6647c87a; no voters: 886bd7a0d91c4ca6947754a55fa13bf8, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:55.034274 20371 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:55.343192 20357 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:55.343303 20357 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:55.343524 20357 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:55.343730 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:55.343765 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:02:55.343976 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8; no voters: 9b54f999704c45bca763910c6647c87a, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:55.344135 20357 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:55.937600 20371 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:55.937731 20371 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:55.937937 20371 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897)
I20260808 07:02:55.938100 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:55.938195 19992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" is_pre_election: true
I20260808 07:02:55.938506 20062 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b54f999704c45bca763910c6647c87a; no voters: 886bd7a0d91c4ca6947754a55fa13bf8, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:56.042436 20357 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:56.042538 20357 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:56.042794 20357 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:56.043004 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:56.043015 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:02:56.043264 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8; no voters: 9b54f999704c45bca763910c6647c87a, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:56.043358 20357 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:56.043475 20418 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=36, attempt_no=0}) took 1449 ms. Trace:
I20260808 07:02:56.043556 20418 rpcz_store.cc:276] 0808 07:02:54.594099 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:54.594140 (+ 41us) service_pool.cc:227] Handling call
0808 07:02:56.043455 (+1449315us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:54.909460 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:54.909476 (+ 16us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:54.909477 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:54.909477 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:54.909491 (+ 14us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:54.909493 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:54.909494 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:54.909505 (+ 11us) tablet.cc:699] Row locks acquired
0808 07:02:54.909506 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:02:54.909513 (+ 7us) write_op.cc:270] Start()
0808 07:02:55.484609 (+575096us) write_op.cc:276] Timestamp: P: 1786172574909469 usec, L: 0
0808 07:02:55.484609 (+ 0us) op_driver.cc:331] REPLICATION: starting
0808 07:02:55.484658 (+ 49us) log.cc:842] Serialized 354 byte log entry
0808 07:02:56.043069 (+558411us) op_driver.cc:444] REPLICATION: finished
0808 07:02:56.043209 (+ 140us) write_op.cc:301] APPLY: starting
0808 07:02:56.043213 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:56.043217 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:56.043220 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:56.043235 (+ 15us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:56.043240 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:56.043241 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:56.043263 (+ 22us) log.cc:842] Serialized 22 byte log entry
0808 07:02:56.043310 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:56.043313 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:56.043314 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":130,"num_ops":1,"prepare.queue_time_us":315232,"prepare.run_cpu_time_us":177,"prepare.run_wall_time_us":575246,"raft.queue_time_us":33,"raft.run_cpu_time_us":35,"raft.run_wall_time_us":35,"replication_time_us":558454,"thread_start_us":65,"threads_started":1}]]}
I20260808 07:02:56.229755 20357 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:56.229866 20357 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:56.230039 20357 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:56.230270 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:56.231489 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:02:56.231753 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8; no voters: 9b54f999704c45bca763910c6647c87a, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:56.231864 20357 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:56.418793 20371 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:56.706944 20357 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:56.707057 20357 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:56.707237 20357 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:56.707573 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:56.707578 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:02:56.707870 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8; no voters: 9b54f999704c45bca763910c6647c87a, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:56.707937 20422 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=37, attempt_no=0}) took 1363 ms. Trace:
I20260808 07:02:56.708012 20357 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:56.707981 20422 rpcz_store.cc:276] 0808 07:02:55.344579 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:55.344611 (+ 32us) service_pool.cc:227] Handling call
0808 07:02:56.707927 (+1363316us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:55.484717 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:55.484732 (+ 15us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:55.484733 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:55.484733 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:55.484746 (+ 13us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:55.484749 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:55.484750 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:55.484783 (+ 33us) tablet.cc:699] Row locks acquired
0808 07:02:55.484784 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:02:55.484791 (+ 7us) write_op.cc:270] Start()
0808 07:02:56.291916 (+807125us) write_op.cc:276] Timestamp: P: 1786172575484729 usec, L: 0
0808 07:02:56.291918 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:02:56.291972 (+ 54us) log.cc:842] Serialized 354 byte log entry
0808 07:02:56.707631 (+415659us) op_driver.cc:444] REPLICATION: finished
0808 07:02:56.707797 (+ 166us) write_op.cc:301] APPLY: starting
0808 07:02:56.707802 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:56.707806 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:56.707809 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:56.707822 (+ 13us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:56.707827 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:56.707828 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:02:56.707878 (+ 50us) log.cc:842] Serialized 43 byte log entry
0808 07:02:56.707897 (+ 19us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:56.707900 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:56.707901 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":154,"num_ops":1,"prepare.queue_time_us":139998,"prepare.run_cpu_time_us":206,"prepare.run_wall_time_us":807278,"replication_time_us":415708,"thread_start_us":75,"threads_started":1}]]}
I20260808 07:02:56.917858 20371 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:56.917958 20371 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:56.918135 20371 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897)
I20260808 07:02:56.918358 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:56.918370 19992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" is_pre_election: true
I20260808 07:02:56.918630 20062 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b54f999704c45bca763910c6647c87a; no voters: 886bd7a0d91c4ca6947754a55fa13bf8, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:56.986253 20357 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:56.986353 20357 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:56.986585 20357 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:56.986771 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:56.986800 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:02:56.987092 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8; no voters: 9b54f999704c45bca763910c6647c87a, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:56.987221 20357 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:57.348496 20357 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:57.348618 20357 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:57.348829 20357 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:57.349129 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:02:57.349210 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:57.349524 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8; no voters: 9b54f999704c45bca763910c6647c87a, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:57.349634 20357 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:57.431993 20371 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:57.934453 20371 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:57.934548 20371 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:57.934814 20371 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897)
I20260808 07:02:57.935029 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:57.935105 19992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" is_pre_election: true
I20260808 07:02:57.935405 20062 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b54f999704c45bca763910c6647c87a; no voters: 886bd7a0d91c4ca6947754a55fa13bf8, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:57.935515 20371 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:58.285835 20357 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:58.285985 20357 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:58.286273 20357 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:02:58.286475 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:58.286525 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:02:58.286777 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8; no voters: 9b54f999704c45bca763910c6647c87a, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:58.732937 20371 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:58.733074 20371 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:58.733265 20371 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897)
I20260808 07:02:58.733403 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:58.733513 19992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" is_pre_election: true
I20260808 07:02:58.733716 20062 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b54f999704c45bca763910c6647c87a; no voters: 886bd7a0d91c4ca6947754a55fa13bf8, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:58.733814 20371 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:58.733959 20435 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=41, attempt_no=0}) took 1384 ms. Trace:
I20260808 07:02:58.734012 20435 rpcz_store.cc:276] 0808 07:02:57.349519 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:57.349542 (+ 23us) service_pool.cc:227] Handling call
0808 07:02:58.733878 (+1384336us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:57.404017 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:57.404033 (+ 16us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:57.404034 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:57.404034 (+ 0us) tablet.cc:660] Decoding operations
0808 07:02:57.404048 (+ 14us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:57.404050 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:57.404051 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:57.404061 (+ 10us) tablet.cc:699] Row locks acquired
0808 07:02:57.404062 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:02:57.404070 (+ 8us) write_op.cc:270] Start()
0808 07:02:58.148205 (+744135us) write_op.cc:276] Timestamp: P: 1786172577404000 usec, L: 0
0808 07:02:58.148206 (+ 1us) op_driver.cc:331] REPLICATION: starting
0808 07:02:58.148296 (+ 90us) log.cc:842] Serialized 354 byte log entry
0808 07:02:58.733501 (+585205us) op_driver.cc:444] REPLICATION: finished
0808 07:02:58.733605 (+ 104us) write_op.cc:301] APPLY: starting
0808 07:02:58.733609 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:58.733614 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:58.733617 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:58.733632 (+ 15us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:58.733637 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:58.733637 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:02:58.733658 (+ 21us) log.cc:842] Serialized 43 byte log entry
0808 07:02:58.733737 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:58.733747 (+ 10us) write_op.cc:454] Released schema lock
0808 07:02:58.733748 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":94,"num_ops":1,"prepare.queue_time_us":54347,"prepare.run_cpu_time_us":242,"prepare.run_wall_time_us":744314,"raft.queue_time_us":16,"raft.run_cpu_time_us":36,"raft.run_wall_time_us":36,"replication_time_us":585291,"thread_start_us":51,"threads_started":1}]]}
I20260808 07:02:58.889367 20357 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:59.982128 20440 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:02:59.982265 20440 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:02:59.982504 20440 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897)
I20260808 07:02:59.982760 19992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 44 } } ignore_live_leader: false dest_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" is_pre_election: true
I20260808 07:02:59.982741 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 44 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:02:59.983016 20445 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=42, attempt_no=0}) took 2046 ms. Trace:
I20260808 07:02:59.983028 20062 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b54f999704c45bca763910c6647c87a; no voters: 886bd7a0d91c4ca6947754a55fa13bf8, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:02:59.983124 20440 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:02:59.983081 20445 rpcz_store.cc:276] 0808 07:02:57.936407 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:57.936429 (+ 22us) service_pool.cc:227] Handling call
0808 07:02:59.983005 (+2046576us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:58.148340 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:58.148355 (+ 15us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:58.148356 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:02:58.148357 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:58.148371 (+ 14us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:58.148373 (+ 2us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:58.148374 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:58.733770 (+585396us) lock_manager.cc:599] Waited 585376us for lock on <redacted> (key hash 44662832435941726)
0808 07:02:58.733773 (+ 3us) tablet.cc:699] Row locks acquired
0808 07:02:58.733774 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:02:58.733784 (+ 10us) write_op.cc:270] Start()
0808 07:02:59.684891 (+951107us) write_op.cc:276] Timestamp: P: 1786172578733755 usec, L: 0
0808 07:02:59.684893 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:02:59.684945 (+ 52us) log.cc:842] Serialized 354 byte log entry
0808 07:02:59.982620 (+297675us) op_driver.cc:444] REPLICATION: finished
0808 07:02:59.982760 (+ 140us) write_op.cc:301] APPLY: starting
0808 07:02:59.982766 (+ 6us) tablet.cc:1365] starting BulkCheckPresence
0808 07:02:59.982771 (+ 5us) tablet.cc:1368] finished BulkCheckPresence
0808 07:02:59.982771 (+ 0us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:02:59.982788 (+ 17us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:02:59.982793 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:02:59.982793 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:02:59.982821 (+ 28us) log.cc:842] Serialized 43 byte log entry
0808 07:02:59.982841 (+ 20us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:02:59.982844 (+ 3us) write_op.cc:454] Released schema lock
0808 07:02:59.982844 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":131,"mutex_wait_us":2,"num_ops":1,"prepare.queue_time_us":211837,"prepare.run_cpu_time_us":244,"prepare.run_wall_time_us":1536664,"raft.queue_time_us":67,"raft.run_cpu_time_us":38,"raft.run_wall_time_us":40,"replication_time_us":297722,"row_lock_wait_count":1,"row_lock_wait_us":585376,"spinlock_wait_cycles":466688,"thread_start_us":76,"threads_started":1}]]}
I20260808 07:03:00.538039 20442 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:03:00.538162 20442 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:03:00.538417 20442 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:03:00.538622 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 44 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:03:00.538630 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 44 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:03:00.538916 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8; no voters: 9b54f999704c45bca763910c6647c87a, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:03:00.974288 20440 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:03:00.974406 20440 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:03:00.974596 20440 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897)
I20260808 07:03:00.974758 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:03:00.974859 19992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" is_pre_election: true
I20260808 07:03:00.975080 20062 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b54f999704c45bca763910c6647c87a; no voters: 886bd7a0d91c4ca6947754a55fa13bf8, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:03:00.975207 20452 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=43, attempt_no=0}) took 2240 ms. Trace:
I20260808 07:03:00.975229 20440 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:03:00.975291 20452 rpcz_store.cc:276] 0808 07:02:58.734592 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:58.734616 (+ 24us) service_pool.cc:227] Handling call
0808 07:03:00.975189 (+2240573us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:02:59.685013 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:02:59.685027 (+ 14us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:02:59.685027 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:02:59.685028 (+ 1us) tablet.cc:660] Decoding operations
0808 07:02:59.685039 (+ 11us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:02:59.685042 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:02:59.685042 (+ 0us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:02:59.685052 (+ 10us) tablet.cc:699] Row locks acquired
0808 07:02:59.685053 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:02:59.685060 (+ 7us) write_op.cc:270] Start()
0808 07:03:00.304173 (+619113us) write_op.cc:276] Timestamp: P: 1786172579685018 usec, L: 0
0808 07:03:00.304174 (+ 1us) op_driver.cc:331] REPLICATION: starting
0808 07:03:00.304233 (+ 59us) log.cc:842] Serialized 354 byte log entry
0808 07:03:00.974878 (+670645us) op_driver.cc:444] REPLICATION: finished
0808 07:03:00.974956 (+ 78us) write_op.cc:301] APPLY: starting
0808 07:03:00.974961 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:03:00.975024 (+ 63us) tablet.cc:1368] finished BulkCheckPresence
0808 07:03:00.975027 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:03:00.975043 (+ 16us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:03:00.975048 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:03:00.975049 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:03:00.975123 (+ 74us) log.cc:842] Serialized 43 byte log entry
0808 07:03:00.975157 (+ 34us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:03:00.975161 (+ 4us) write_op.cc:454] Released schema lock
0808 07:03:00.975161 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":70,"num_ops":1,"prepare.queue_time_us":950318,"prepare.run_cpu_time_us":194,"prepare.run_wall_time_us":619268,"replication_time_us":670699,"thread_start_us":42,"threads_started":1}]]}
I20260808 07:03:01.387194 20442 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:03:01.434279 20442 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:03:01.434391 20442 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:03:01.434592 20442 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:03:01.434793 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:03:01.434829 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:03:01.435048 20452 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=44, attempt_no=0}) took 1451 ms. Trace:
I20260808 07:03:01.435102 20452 rpcz_store.cc:276] 0808 07:02:59.983837 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:02:59.983880 (+ 43us) service_pool.cc:227] Handling call
0808 07:03:01.435042 (+1451162us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:03:00.304292 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:03:00.304304 (+ 12us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:03:00.304305 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:03:00.304305 (+ 0us) tablet.cc:660] Decoding operations
0808 07:03:00.304316 (+ 11us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:03:00.304319 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:03:00.304319 (+ 0us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:03:00.304328 (+ 9us) tablet.cc:699] Row locks acquired
0808 07:03:00.304328 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:03:00.304336 (+ 8us) write_op.cc:270] Start()
0808 07:03:00.975459 (+671123us) write_op.cc:276] Timestamp: P: 1786172580304285 usec, L: 0
0808 07:03:00.975460 (+ 1us) op_driver.cc:331] REPLICATION: starting
0808 07:03:00.975505 (+ 45us) log.cc:842] Serialized 354 byte log entry
0808 07:03:01.434904 (+459399us) op_driver.cc:444] REPLICATION: finished
0808 07:03:01.434946 (+ 42us) write_op.cc:301] APPLY: starting
0808 07:03:01.434950 (+ 4us) tablet.cc:1365] starting BulkCheckPresence
0808 07:03:01.434954 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:03:01.434956 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:03:01.434970 (+ 14us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:03:01.434975 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:03:01.434976 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:03:01.434999 (+ 23us) log.cc:842] Serialized 43 byte log entry
0808 07:03:01.435014 (+ 15us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:03:01.435017 (+ 3us) write_op.cc:454] Released schema lock
0808 07:03:01.435017 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":30,"num_ops":1,"prepare.queue_time_us":320327,"prepare.run_cpu_time_us":175,"prepare.run_wall_time_us":671232,"replication_time_us":459439}]]}
I20260808 07:03:01.435473 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8; no voters: 9b54f999704c45bca763910c6647c87a, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:03:01.435598 20442 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:03:01.836594 20440 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:03:01.836736 20440 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:03:01.836985 20440 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897)
I20260808 07:03:01.837258 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:03:01.837271 19992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" is_pre_election: true
I20260808 07:03:01.837556 20062 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b54f999704c45bca763910c6647c87a; no voters: 886bd7a0d91c4ca6947754a55fa13bf8, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:03:02.029193 20442 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:03:02.029318 20442 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:03:02.029560 20442 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:03:02.029706 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" is_pre_election: true
I20260808 07:03:02.029772 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:03:02.030026 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8; no voters: 9b54f999704c45bca763910c6647c87a, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:03:02.030252 20460 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50484 (ReqId={client: e806fae4a5bd40fabcb2551d54ab8a66, seq_no=45, attempt_no=0}) took 1054 ms. Trace:
I20260808 07:03:02.030290 20442 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:03:02.030376 20460 rpcz_store.cc:276] 0808 07:03:00.976042 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:03:00.976072 (+ 30us) service_pool.cc:227] Handling call
0808 07:03:02.030242 (+1054170us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:03:00.976189 (+ 0us) write_op.cc:183] PREPARE: starting on tablet edea628254264454bbd00d793271a973
0808 07:03:00.976199 (+ 10us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:03:00.976200 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:03:00.976200 (+ 0us) tablet.cc:660] Decoding operations
0808 07:03:00.976208 (+ 8us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:03:00.976209 (+ 1us) write_op.cc:641] Partition lock acquired for write op
0808 07:03:00.976210 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:03:00.976217 (+ 7us) tablet.cc:699] Row locks acquired
0808 07:03:00.976217 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:03:00.976221 (+ 4us) write_op.cc:270] Start()
0808 07:03:01.474363 (+498142us) write_op.cc:276] Timestamp: P: 1786172580976177 usec, L: 0
0808 07:03:01.474365 (+ 2us) op_driver.cc:331] REPLICATION: starting
0808 07:03:01.474432 (+ 67us) log.cc:842] Serialized 354 byte log entry
0808 07:03:02.029865 (+555433us) op_driver.cc:444] REPLICATION: finished
0808 07:03:02.029995 (+ 130us) write_op.cc:301] APPLY: starting
0808 07:03:02.030000 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:03:02.030004 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:03:02.030006 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:03:02.030019 (+ 13us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:03:02.030024 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:03:02.030024 (+ 0us) write_op.cc:312] APPLY: finished
0808 07:03:02.030043 (+ 19us) log.cc:842] Serialized 22 byte log entry
0808 07:03:02.030158 (+ 115us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:03:02.030215 (+ 57us) write_op.cc:454] Released schema lock
0808 07:03:02.030216 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":121,"num_ops":1,"prepare.queue_time_us":38,"prepare.run_cpu_time_us":195,"prepare.run_wall_time_us":498294,"raft.queue_time_us":36,"raft.run_cpu_time_us":59,"raft.run_wall_time_us":59,"replication_time_us":555496,"thread_start_us":39,"threads_started":1}]]}
I20260808 07:03:02.070012 19970 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260808 07:03:02.070431 19830 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260808 07:03:02.280728 20440 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:03:02.281993 20112 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
39f0b8ec22d44894b20ce12b523a6a8e | 127.15.250.190:43327 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.250.148:39425 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
886bd7a0d91c4ca6947754a55fa13bf8 | 127.15.250.130:43897 | HEALTHY | <none> | 0 | 0
9b54f999704c45bca763910c6647c87a | 127.15.250.131:44861 | HEALTHY | <none> | 0 | 0
d20f18f45fb644f3a9741e9dc33f4dea | 127.15.250.129:38229 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
inject_latency_ms_before_starting_op | 1000 | hidden,advanced | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.15.250.129 | experimental | 127.15.250.129:38229
local_ip_for_outbound_sockets | 127.15.250.130 | experimental | 127.15.250.130:43897
local_ip_for_outbound_sockets | 127.15.250.131 | experimental | 127.15.250.131:44861
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.15.250.129:38229
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.15.250.130:43897
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.15.250.131:44861
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.250.148:39425 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260808 07:03:02.290611 16362 log_verifier.cc:126] Checking tablet edea628254264454bbd00d793271a973
I20260808 07:03:02.313599 16362 log_verifier.cc:177] Verified matching terms for 47 ops in tablet edea628254264454bbd00d793271a973
I20260808 07:03:02.326992 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 19764
I20260808 07:03:02.332655 20440 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:03:02.332758 20440 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:03:02.332927 20440 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897)
I20260808 07:03:02.332958 20442 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:03:02.333035 20442 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:03:02.333184 20442 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 9b54f999704c45bca763910c6647c87a (127.15.250.131:44861)
I20260808 07:03:02.333377 20133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "9b54f999704c45bca763910c6647c87a" is_pre_election: true
I20260808 07:03:02.333457 20133 raft_consensus.cc:2454] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 886bd7a0d91c4ca6947754a55fa13bf8 in term 1.
I20260808 07:03:02.333575 19991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edea628254264454bbd00d793271a973" candidate_uuid: "9b54f999704c45bca763910c6647c87a" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" is_pre_election: true
I20260808 07:03:02.333595 19923 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8, 9b54f999704c45bca763910c6647c87a; no voters:
I20260808 07:03:02.333673 19991 raft_consensus.cc:2454] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b54f999704c45bca763910c6647c87a in term 1.
I20260808 07:03:02.333818 20442 raft_consensus.cc:2781] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 07:03:02.333859 20442 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:03:02.333849 20062 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 886bd7a0d91c4ca6947754a55fa13bf8, 9b54f999704c45bca763910c6647c87a; no voters:
I20260808 07:03:02.333882 20442 raft_consensus.cc:3037] T edea628254264454bbd00d793271a973 P 886bd7a0d91c4ca6947754a55fa13bf8 [term 1 FOLLOWER]: Advancing to term 2
W20260808 07:03:02.333947 20062 connection.cc:535] client connection to 127.15.250.129:38229 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 07:03:02.333986 20062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 1 similar messages]
W20260808 07:03:02.334188 19923 connection.cc:535] client connection to 127.15.250.129:38229 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 07:03:02.334249 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 19905
W20260808 07:03:02.334267 19923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 07:03:02.337628 20440 raft_consensus.cc:2781] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 07:03:02.337699 20440 raft_consensus.cc:482] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Starting leader election (detected failure of leader d20f18f45fb644f3a9741e9dc33f4dea)
I20260808 07:03:02.337728 20440 raft_consensus.cc:3037] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 1 FOLLOWER]: Advancing to term 2
W20260808 07:03:02.338240 20062 leader_election.cc:334] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229): Network error: Client connection negotiation failed: client connection to 127.15.250.129:38229: connect: Connection refused (error 111)
I20260808 07:03:02.338601 20440 raft_consensus.cc:504] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f18f45fb644f3a9741e9dc33f4dea" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 38229 } } peers { permanent_uuid: "886bd7a0d91c4ca6947754a55fa13bf8" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 43897 } } peers { permanent_uuid: "9b54f999704c45bca763910c6647c87a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 44861 } }
I20260808 07:03:02.338745 20440 leader_election.cc:288] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 election: Requested vote from peers d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229), 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897)
W20260808 07:03:02.339211 20062 leader_election.cc:334] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d20f18f45fb644f3a9741e9dc33f4dea (127.15.250.129:38229): Network error: Client connection negotiation failed: client connection to 127.15.250.129:38229: connect: Connection refused (error 111)
W20260808 07:03:02.342082 20062 connection.cc:535] client connection to 127.15.250.130:43897 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 07:03:02.342504 20062 leader_election.cc:334] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 886bd7a0d91c4ca6947754a55fa13bf8 (127.15.250.130:43897): Network error: Client connection negotiation failed: client connection to 127.15.250.130:43897: connect: Connection refused (error 111)
I20260808 07:03:02.342576 20062 leader_election.cc:302] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b54f999704c45bca763910c6647c87a; no voters: 886bd7a0d91c4ca6947754a55fa13bf8, d20f18f45fb644f3a9741e9dc33f4dea
I20260808 07:03:02.342656 20440 raft_consensus.cc:2726] T edea628254264454bbd00d793271a973 P 9b54f999704c45bca763910c6647c87a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 07:03:02.342656 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 20045
I20260808 07:03:02.350347 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 19665
2026-08-08T07:03:02Z chronyd exiting
[ OK ] RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys (31762 ms)
[ RUN ] RaftConsensusElectionITest.AutomaticLeaderElection
I20260808 07:03:02.373701 16362 test_util.cc:268] Using random seed: -1236817601
I20260808 07:03:02.374245 16362 ts_itest-base.cc:112] Starting cluster with:
I20260808 07:03:02.374302 16362 ts_itest-base.cc:113] --------------
I20260808 07:03:02.374321 16362 ts_itest-base.cc:114] 5 tablet servers
I20260808 07:03:02.374348 16362 ts_itest-base.cc:115] 5 replicas per TS
I20260808 07:03:02.374373 16362 ts_itest-base.cc:116] --------------
2026-08-08T07:03:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:03:02Z Disabled control of system clock
I20260808 07:03:02.383477 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.250.190:32917
--webserver_interface=127.15.250.190
--webserver_port=0
--builtin_ntp_servers=127.15.250.148:45891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.250.190:32917 with env {}
W20260808 07:03:02.468406 20509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:02.468641 20509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:02.468676 20509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:02.469838 20509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:03:02.469904 20509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:02.469928 20509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:03:02.469950 20509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:03:02.471311 20509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:45891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.250.190:32917
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.250.190:32917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.250.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.20509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:02.471596 20509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:02.471913 20509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:02.477506 20515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:02.477564 20518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:02.477715 20509 server_base.cc:1034] running on GCE node
W20260808 07:03:02.477522 20516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:02.478160 20509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:02.478530 20509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:02.479684 20509 hybrid_clock.cc:648] HybridClock initialized: now 1786172582479665 us; error 42 us; skew 500 ppm
I20260808 07:03:02.480965 20509 webserver.cc:460] Webserver started at http://127.15.250.190:42983/ using document root <none> and password file <none>
I20260808 07:03:02.481190 20509 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:02.481233 20509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:02.481333 20509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:02.482285 20509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "4fdbc1f474314637a592745bd0faeee9"
format_stamp: "Formatted at 2026-08-08 07:03:02 on dist-test-slave-0b58"
I20260808 07:03:02.482591 20509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "4fdbc1f474314637a592745bd0faeee9"
format_stamp: "Formatted at 2026-08-08 07:03:02 on dist-test-slave-0b58"
I20260808 07:03:02.483804 20509 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 07:03:02.484525 20526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:02.484701 20509 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 07:03:02.484800 20509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4fdbc1f474314637a592745bd0faeee9"
format_stamp: "Formatted at 2026-08-08 07:03:02 on dist-test-slave-0b58"
I20260808 07:03:02.484855 20509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:02.493534 20509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:02.493813 20509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:02.493917 20509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:02.497900 20509 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.190:32917
I20260808 07:03:02.497931 20579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.190:32917 every 8 connection(s)
I20260808 07:03:02.498294 20509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260808 07:03:02.498754 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 20509
I20260808 07:03:02.498847 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260808 07:03:02.499104 20580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:02.502609 20580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fdbc1f474314637a592745bd0faeee9: Bootstrap starting.
I20260808 07:03:02.503391 20580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fdbc1f474314637a592745bd0faeee9: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:02.503707 20580 log.cc:824] T 00000000000000000000000000000000 P 4fdbc1f474314637a592745bd0faeee9: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:02.504516 20580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fdbc1f474314637a592745bd0faeee9: No bootstrap required, opened a new log
I20260808 07:03:02.506526 20580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fdbc1f474314637a592745bd0faeee9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fdbc1f474314637a592745bd0faeee9" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 32917 } }
I20260808 07:03:02.506665 20580 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fdbc1f474314637a592745bd0faeee9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:03:02.506690 20580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fdbc1f474314637a592745bd0faeee9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fdbc1f474314637a592745bd0faeee9, State: Initialized, Role: FOLLOWER
I20260808 07:03:02.506811 20580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fdbc1f474314637a592745bd0faeee9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fdbc1f474314637a592745bd0faeee9" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 32917 } }
I20260808 07:03:02.506886 20580 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fdbc1f474314637a592745bd0faeee9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:03:02.506908 20580 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fdbc1f474314637a592745bd0faeee9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:03:02.506937 20580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fdbc1f474314637a592745bd0faeee9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:02.507440 20580 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fdbc1f474314637a592745bd0faeee9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fdbc1f474314637a592745bd0faeee9" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 32917 } }
I20260808 07:03:02.507530 20580 leader_election.cc:302] T 00000000000000000000000000000000 P 4fdbc1f474314637a592745bd0faeee9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fdbc1f474314637a592745bd0faeee9; no voters:
I20260808 07:03:02.507656 20580 leader_election.cc:288] T 00000000000000000000000000000000 P 4fdbc1f474314637a592745bd0faeee9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:03:02.507730 20585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fdbc1f474314637a592745bd0faeee9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:02.507882 20585 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fdbc1f474314637a592745bd0faeee9 [term 1 LEADER]: Becoming Leader. State: Replica: 4fdbc1f474314637a592745bd0faeee9, State: Running, Role: LEADER
I20260808 07:03:02.507970 20580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fdbc1f474314637a592745bd0faeee9 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:03:02.508023 20585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fdbc1f474314637a592745bd0faeee9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fdbc1f474314637a592745bd0faeee9" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 32917 } }
I20260808 07:03:02.508463 20586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fdbc1f474314637a592745bd0faeee9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fdbc1f474314637a592745bd0faeee9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fdbc1f474314637a592745bd0faeee9" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 32917 } } }
I20260808 07:03:02.508476 20587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fdbc1f474314637a592745bd0faeee9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fdbc1f474314637a592745bd0faeee9. Latest consensus state: current_term: 1 leader_uuid: "4fdbc1f474314637a592745bd0faeee9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fdbc1f474314637a592745bd0faeee9" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 32917 } } }
I20260808 07:03:02.508680 20586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fdbc1f474314637a592745bd0faeee9 [sys.catalog]: This master's current role is: LEADER
I20260808 07:03:02.508695 20587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fdbc1f474314637a592745bd0faeee9 [sys.catalog]: This master's current role is: LEADER
I20260808 07:03:02.509023 20595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:03:02.509529 20595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:03:02.511106 20595 catalog_manager.cc:1347] Generated new cluster ID: d86a36b181f647d3a8cf5f89749dcaa8
I20260808 07:03:02.511178 20595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:03:02.521564 20595 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:03:02.522200 20595 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:03:02.528138 20595 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fdbc1f474314637a592745bd0faeee9: Generated new TSK 0
I20260808 07:03:02.528383 20595 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:03:02.530893 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.129:0
--local_ip_for_outbound_sockets=127.15.250.129
--webserver_interface=127.15.250.129
--webserver_port=0
--tserver_master_addrs=127.15.250.190:32917
--builtin_ntp_servers=127.15.250.148:45891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
W20260808 07:03:02.632655 20605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:02.632863 20605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:02.632885 20605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:02.634037 20605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:02.634087 20605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.129
I20260808 07:03:02.635390 20605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:45891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.250.129
--webserver_port=0
--tserver_master_addrs=127.15.250.190:32917
--never_fsync=true
--heap_profile_path=/tmp/kudu.20605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.129
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:02.635627 20605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:02.635949 20605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:02.641996 20611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:02.642009 20613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:02.642017 20610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:02.642253 20605 server_base.cc:1034] running on GCE node
I20260808 07:03:02.642529 20605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:02.642804 20605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:02.643973 20605 hybrid_clock.cc:648] HybridClock initialized: now 1786172582643946 us; error 46 us; skew 500 ppm
I20260808 07:03:02.645375 20605 webserver.cc:460] Webserver started at http://127.15.250.129:46843/ using document root <none> and password file <none>
I20260808 07:03:02.645650 20605 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:02.645725 20605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:02.645907 20605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:02.647002 20605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd"
format_stamp: "Formatted at 2026-08-08 07:03:02 on dist-test-slave-0b58"
I20260808 07:03:02.647362 20605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd"
format_stamp: "Formatted at 2026-08-08 07:03:02 on dist-test-slave-0b58"
I20260808 07:03:02.648852 20605 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 07:03:02.649711 20620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:02.649930 20605 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:03:02.650031 20605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd"
format_stamp: "Formatted at 2026-08-08 07:03:02 on dist-test-slave-0b58"
I20260808 07:03:02.650118 20605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:02.677716 20605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:02.678025 20605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:02.678130 20605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:02.678328 20605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:03:02.678681 20605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:03:02.678740 20605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:02.678793 20605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:03:02.678826 20605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:02.684880 20605 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.129:34097
I20260808 07:03:02.684963 20740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.129:34097 every 8 connection(s)
I20260808 07:03:02.685324 20605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260808 07:03:02.687532 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 20605
I20260808 07:03:02.687662 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260808 07:03:02.688975 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.130:0
--local_ip_for_outbound_sockets=127.15.250.130
--webserver_interface=127.15.250.130
--webserver_port=0
--tserver_master_addrs=127.15.250.190:32917
--builtin_ntp_servers=127.15.250.148:45891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20260808 07:03:02.692396 20741 heartbeater.cc:347] Connected to a master server at 127.15.250.190:32917
I20260808 07:03:02.692516 20741 heartbeater.cc:464] Registering TS with master...
I20260808 07:03:02.692719 20741 heartbeater.cc:511] Master 127.15.250.190:32917 requested a full tablet report, sending...
I20260808 07:03:02.693297 20543 ts_manager.cc:194] Registered new tserver with Master: 4ede38c8f2ec4f82b36b8f7aa57809bd (127.15.250.129:34097)
I20260808 07:03:02.694003 20543 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.129:40297
W20260808 07:03:02.773866 20744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:02.774087 20744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:02.774109 20744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:02.775436 20744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:02.775509 20744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.130
I20260808 07:03:02.776960 20744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:45891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.250.130
--webserver_port=0
--tserver_master_addrs=127.15.250.190:32917
--never_fsync=true
--heap_profile_path=/tmp/kudu.20744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.130
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:02.777165 20744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:02.777437 20744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 07:03:02.782724 20744 server_base.cc:1034] running on GCE node
W20260808 07:03:02.782686 20750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:02.782693 20753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:02.782693 20751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:02.783111 20744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:02.783378 20744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:02.784554 20744 hybrid_clock.cc:648] HybridClock initialized: now 1786172582784528 us; error 43 us; skew 500 ppm
I20260808 07:03:02.786029 20744 webserver.cc:460] Webserver started at http://127.15.250.130:38385/ using document root <none> and password file <none>
I20260808 07:03:02.786301 20744 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:02.786384 20744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:02.786537 20744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:02.787678 20744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "da18746c3d9d4ad0844c8f3c9197478f"
format_stamp: "Formatted at 2026-08-08 07:03:02 on dist-test-slave-0b58"
I20260808 07:03:02.788089 20744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "da18746c3d9d4ad0844c8f3c9197478f"
format_stamp: "Formatted at 2026-08-08 07:03:02 on dist-test-slave-0b58"
I20260808 07:03:02.789588 20744 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 07:03:02.790432 20760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:02.790616 20744 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 07:03:02.790728 20744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "da18746c3d9d4ad0844c8f3c9197478f"
format_stamp: "Formatted at 2026-08-08 07:03:02 on dist-test-slave-0b58"
I20260808 07:03:02.790804 20744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:02.809491 20744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:02.809865 20744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:02.810017 20744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:02.810283 20744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:03:02.810647 20744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:03:02.810706 20744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:02.810740 20744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:03:02.810767 20744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:02.816406 20744 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.130:37825
I20260808 07:03:02.816450 20879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.130:37825 every 8 connection(s)
I20260808 07:03:02.816820 20744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260808 07:03:02.822572 20880 heartbeater.cc:347] Connected to a master server at 127.15.250.190:32917
I20260808 07:03:02.822664 20880 heartbeater.cc:464] Registering TS with master...
I20260808 07:03:02.822853 20880 heartbeater.cc:511] Master 127.15.250.190:32917 requested a full tablet report, sending...
I20260808 07:03:02.823304 20543 ts_manager.cc:194] Registered new tserver with Master: da18746c3d9d4ad0844c8f3c9197478f (127.15.250.130:37825)
I20260808 07:03:02.823661 20543 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.130:37887
I20260808 07:03:02.825498 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 20744
I20260808 07:03:02.825634 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260808 07:03:02.826771 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.131:0
--local_ip_for_outbound_sockets=127.15.250.131
--webserver_interface=127.15.250.131
--webserver_port=0
--tserver_master_addrs=127.15.250.190:32917
--builtin_ntp_servers=127.15.250.148:45891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
W20260808 07:03:02.909950 20883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:02.910164 20883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:02.910184 20883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:02.911373 20883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:02.911433 20883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.131
I20260808 07:03:02.912811 20883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:45891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.250.131
--webserver_port=0
--tserver_master_addrs=127.15.250.190:32917
--never_fsync=true
--heap_profile_path=/tmp/kudu.20883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.131
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:02.913081 20883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:02.913408 20883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:02.918819 20889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:02.918839 20892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:02.918882 20890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:02.918879 20883 server_base.cc:1034] running on GCE node
I20260808 07:03:02.919245 20883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:02.919459 20883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:02.920603 20883 hybrid_clock.cc:648] HybridClock initialized: now 1786172582920593 us; error 33 us; skew 500 ppm
I20260808 07:03:02.921945 20883 webserver.cc:460] Webserver started at http://127.15.250.131:42597/ using document root <none> and password file <none>
I20260808 07:03:02.922171 20883 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:02.922214 20883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:02.922308 20883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:02.923328 20883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e37990d179424f26b5dae70488a5c8ef"
format_stamp: "Formatted at 2026-08-08 07:03:02 on dist-test-slave-0b58"
I20260808 07:03:02.923642 20883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e37990d179424f26b5dae70488a5c8ef"
format_stamp: "Formatted at 2026-08-08 07:03:02 on dist-test-slave-0b58"
I20260808 07:03:02.925124 20883 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 07:03:02.926002 20899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:02.926321 20883 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 07:03:02.926437 20883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e37990d179424f26b5dae70488a5c8ef"
format_stamp: "Formatted at 2026-08-08 07:03:02 on dist-test-slave-0b58"
I20260808 07:03:02.926517 20883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:02.938135 20883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:02.938484 20883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:02.938635 20883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:02.938872 20883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:03:02.939210 20883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:03:02.939270 20883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:02.939313 20883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:03:02.939338 20883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:02.945447 20883 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.131:40561
I20260808 07:03:02.945490 21017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.131:40561 every 8 connection(s)
I20260808 07:03:02.945868 20883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260808 07:03:02.951576 21018 heartbeater.cc:347] Connected to a master server at 127.15.250.190:32917
I20260808 07:03:02.951685 21018 heartbeater.cc:464] Registering TS with master...
I20260808 07:03:02.951897 21018 heartbeater.cc:511] Master 127.15.250.190:32917 requested a full tablet report, sending...
I20260808 07:03:02.952353 20543 ts_manager.cc:194] Registered new tserver with Master: e37990d179424f26b5dae70488a5c8ef (127.15.250.131:40561)
I20260808 07:03:02.952595 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 20883
I20260808 07:03:02.952706 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260808 07:03:02.952896 20543 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.131:47445
I20260808 07:03:02.953847 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.132:0
--local_ip_for_outbound_sockets=127.15.250.132
--webserver_interface=127.15.250.132
--webserver_port=0
--tserver_master_addrs=127.15.250.190:32917
--builtin_ntp_servers=127.15.250.148:45891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
W20260808 07:03:03.037381 21022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:03.037591 21022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:03.037612 21022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:03.038787 21022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:03.038851 21022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.132
I20260808 07:03:03.040097 21022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:45891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.15.250.132
--webserver_port=0
--tserver_master_addrs=127.15.250.190:32917
--never_fsync=true
--heap_profile_path=/tmp/kudu.21022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.132
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:03.040354 21022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:03.040728 21022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:03.045941 21028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:03.046033 21027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:03.045919 21030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:03.046144 21022 server_base.cc:1034] running on GCE node
I20260808 07:03:03.046468 21022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:03.046747 21022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:03.047930 21022 hybrid_clock.cc:648] HybridClock initialized: now 1786172583047921 us; error 40 us; skew 500 ppm
I20260808 07:03:03.049422 21022 webserver.cc:460] Webserver started at http://127.15.250.132:36625/ using document root <none> and password file <none>
I20260808 07:03:03.049700 21022 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:03.049773 21022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:03.049923 21022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:03.050942 21022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4"
format_stamp: "Formatted at 2026-08-08 07:03:03 on dist-test-slave-0b58"
I20260808 07:03:03.051308 21022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4"
format_stamp: "Formatted at 2026-08-08 07:03:03 on dist-test-slave-0b58"
I20260808 07:03:03.052865 21022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 07:03:03.053826 21038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:03.054018 21022 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 07:03:03.054116 21022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4"
format_stamp: "Formatted at 2026-08-08 07:03:03 on dist-test-slave-0b58"
I20260808 07:03:03.054200 21022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:03.063805 21022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:03.064108 21022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:03.064322 21022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:03.064558 21022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:03:03.064893 21022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:03:03.064949 21022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:03.065011 21022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:03:03.065037 21022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:03.070639 21022 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.132:46195
I20260808 07:03:03.070678 21156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.132:46195 every 8 connection(s)
I20260808 07:03:03.071048 21022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20260808 07:03:03.076779 21157 heartbeater.cc:347] Connected to a master server at 127.15.250.190:32917
I20260808 07:03:03.076872 21157 heartbeater.cc:464] Registering TS with master...
I20260808 07:03:03.077083 21157 heartbeater.cc:511] Master 127.15.250.190:32917 requested a full tablet report, sending...
I20260808 07:03:03.077461 20543 ts_manager.cc:194] Registered new tserver with Master: 3a0f4ebf6ab9468898d8d6f756cc81f4 (127.15.250.132:46195)
I20260808 07:03:03.077834 20543 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.132:34713
I20260808 07:03:03.079200 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 21022
I20260808 07:03:03.079265 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20260808 07:03:03.082173 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.133:0
--local_ip_for_outbound_sockets=127.15.250.133
--webserver_interface=127.15.250.133
--webserver_port=0
--tserver_master_addrs=127.15.250.190:32917
--builtin_ntp_servers=127.15.250.148:45891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
W20260808 07:03:03.165339 21160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:03.165555 21160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:03.165576 21160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:03.166755 21160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:03.166814 21160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.133
I20260808 07:03:03.168219 21160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:45891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.15.250.133
--webserver_port=0
--tserver_master_addrs=127.15.250.190:32917
--never_fsync=true
--heap_profile_path=/tmp/kudu.21160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.133
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:03.168556 21160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:03.168895 21160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:03.174233 21168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:03.174240 21165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:03.174257 21166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:03.174619 21160 server_base.cc:1034] running on GCE node
I20260808 07:03:03.174870 21160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:03.175143 21160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:03.176327 21160 hybrid_clock.cc:648] HybridClock initialized: now 1786172583176294 us; error 54 us; skew 500 ppm
I20260808 07:03:03.177745 21160 webserver.cc:460] Webserver started at http://127.15.250.133:36797/ using document root <none> and password file <none>
I20260808 07:03:03.178012 21160 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:03.178061 21160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:03.178205 21160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:03.179250 21160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "740b28ba0e48425a9279f6f51684e9f3"
format_stamp: "Formatted at 2026-08-08 07:03:03 on dist-test-slave-0b58"
I20260808 07:03:03.179636 21160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "740b28ba0e48425a9279f6f51684e9f3"
format_stamp: "Formatted at 2026-08-08 07:03:03 on dist-test-slave-0b58"
I20260808 07:03:03.181084 21160 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 07:03:03.181980 21174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:03.182196 21160 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:03:03.182274 21160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "740b28ba0e48425a9279f6f51684e9f3"
format_stamp: "Formatted at 2026-08-08 07:03:03 on dist-test-slave-0b58"
I20260808 07:03:03.182382 21160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:03.199555 21160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:03.199892 21160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:03.200039 21160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:03.200368 21160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:03:03.200718 21160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:03:03.200752 21160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:03.200774 21160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:03:03.200786 21160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:03.206991 21160 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.133:33829
I20260808 07:03:03.207034 21291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.133:33829 every 8 connection(s)
I20260808 07:03:03.207494 21160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20260808 07:03:03.207902 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 21160
I20260808 07:03:03.208017 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20260808 07:03:03.213613 21292 heartbeater.cc:347] Connected to a master server at 127.15.250.190:32917
I20260808 07:03:03.213713 21292 heartbeater.cc:464] Registering TS with master...
I20260808 07:03:03.213862 21292 heartbeater.cc:511] Master 127.15.250.190:32917 requested a full tablet report, sending...
I20260808 07:03:03.214197 20543 ts_manager.cc:194] Registered new tserver with Master: 740b28ba0e48425a9279f6f51684e9f3 (127.15.250.133:33829)
I20260808 07:03:03.214579 20543 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.133:58997
I20260808 07:03:03.220032 16362 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20260808 07:03:03.226588 20543 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33370:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20260808 07:03:03.228153 20543 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 6 tablet servers would be needed, 5 are available. Consider bringing up more tablet servers.
I20260808 07:03:03.236445 20947 tablet_service.cc:1467] Processing CreateTablet for tablet c36a91742d9144c58e84540f45c52a7c (DEFAULT_TABLE table=TestTable [id=63b6984a45ff42c08ed80fd16041361f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:03:03.236614 20669 tablet_service.cc:1467] Processing CreateTablet for tablet c36a91742d9144c58e84540f45c52a7c (DEFAULT_TABLE table=TestTable [id=63b6984a45ff42c08ed80fd16041361f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:03:03.236889 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c36a91742d9144c58e84540f45c52a7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:03.236989 20669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c36a91742d9144c58e84540f45c52a7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:03.237130 20812 tablet_service.cc:1467] Processing CreateTablet for tablet c36a91742d9144c58e84540f45c52a7c (DEFAULT_TABLE table=TestTable [id=63b6984a45ff42c08ed80fd16041361f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:03:03.237447 20812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c36a91742d9144c58e84540f45c52a7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:03.240247 21313 tablet_bootstrap.cc:492] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef: Bootstrap starting.
I20260808 07:03:03.240733 21224 tablet_service.cc:1467] Processing CreateTablet for tablet c36a91742d9144c58e84540f45c52a7c (DEFAULT_TABLE table=TestTable [id=63b6984a45ff42c08ed80fd16041361f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:03:03.240994 21313 tablet_bootstrap.cc:654] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:03.240897 21087 tablet_service.cc:1467] Processing CreateTablet for tablet c36a91742d9144c58e84540f45c52a7c (DEFAULT_TABLE table=TestTable [id=63b6984a45ff42c08ed80fd16041361f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:03:03.241112 21224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c36a91742d9144c58e84540f45c52a7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:03.241237 21087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c36a91742d9144c58e84540f45c52a7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:03.241485 21313 log.cc:824] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:03.242280 21313 tablet_bootstrap.cc:492] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef: No bootstrap required, opened a new log
I20260808 07:03:03.242341 21313 ts_tablet_manager.cc:1397] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 07:03:03.243801 21315 tablet_bootstrap.cc:492] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4: Bootstrap starting.
I20260808 07:03:03.244517 21315 tablet_bootstrap.cc:654] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:03.244534 21316 tablet_bootstrap.cc:492] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f: Bootstrap starting.
I20260808 07:03:03.244905 21315 log.cc:824] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:03.245286 21316 tablet_bootstrap.cc:654] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:03.245270 21313 raft_consensus.cc:348] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 } } peers { permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 } } peers { permanent_uuid: "e37990d179424f26b5dae70488a5c8ef" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 40561 } } peers { permanent_uuid: "740b28ba0e48425a9279f6f51684e9f3" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 33829 } } peers { permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 } }
I20260808 07:03:03.245469 21313 raft_consensus.cc:374] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:03:03.245514 21313 raft_consensus.cc:729] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e37990d179424f26b5dae70488a5c8ef, State: Initialized, Role: FOLLOWER
I20260808 07:03:03.245688 21316 log.cc:824] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:03.245674 21313 consensus_queue.cc:260] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 } } peers { permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 } } peers { permanent_uuid: "e37990d179424f26b5dae70488a5c8ef" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 40561 } } peers { permanent_uuid: "740b28ba0e48425a9279f6f51684e9f3" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 33829 } } peers { permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 } }
I20260808 07:03:03.245903 21317 tablet_bootstrap.cc:492] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd: Bootstrap starting.
I20260808 07:03:03.245960 21313 ts_tablet_manager.cc:1428] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260808 07:03:03.246075 21018 heartbeater.cc:503] Master 127.15.250.190:32917 was elected leader, sending a full tablet report...
I20260808 07:03:03.246292 21319 tablet_bootstrap.cc:492] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3: Bootstrap starting.
I20260808 07:03:03.246510 21316 tablet_bootstrap.cc:492] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f: No bootstrap required, opened a new log
I20260808 07:03:03.246575 21316 ts_tablet_manager.cc:1397] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 07:03:03.246709 21315 tablet_bootstrap.cc:492] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4: No bootstrap required, opened a new log
I20260808 07:03:03.246763 21315 ts_tablet_manager.cc:1397] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260808 07:03:03.246935 21319 tablet_bootstrap.cc:654] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:03.247283 21319 log.cc:824] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:03.248047 21317 tablet_bootstrap.cc:654] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:03.248349 21317 log.cc:824] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:03.248973 21317 tablet_bootstrap.cc:492] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd: No bootstrap required, opened a new log
I20260808 07:03:03.249019 21317 ts_tablet_manager.cc:1397] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260808 07:03:03.249408 21316 raft_consensus.cc:348] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 } } peers { permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 } } peers { permanent_uuid: "e37990d179424f26b5dae70488a5c8ef" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 40561 } } peers { permanent_uuid: "740b28ba0e48425a9279f6f51684e9f3" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 33829 } } peers { permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 } }
I20260808 07:03:03.249408 21315 raft_consensus.cc:348] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 } } peers { permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 } } peers { permanent_uuid: "e37990d179424f26b5dae70488a5c8ef" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 40561 } } peers { permanent_uuid: "740b28ba0e48425a9279f6f51684e9f3" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 33829 } } peers { permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 } }
I20260808 07:03:03.249571 21315 raft_consensus.cc:374] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:03:03.249571 21316 raft_consensus.cc:374] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:03:03.249607 21315 raft_consensus.cc:729] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a0f4ebf6ab9468898d8d6f756cc81f4, State: Initialized, Role: FOLLOWER
I20260808 07:03:03.249611 21316 raft_consensus.cc:729] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da18746c3d9d4ad0844c8f3c9197478f, State: Initialized, Role: FOLLOWER
I20260808 07:03:03.249745 21316 consensus_queue.cc:260] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 } } peers { permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 } } peers { permanent_uuid: "e37990d179424f26b5dae70488a5c8ef" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 40561 } } peers { permanent_uuid: "740b28ba0e48425a9279f6f51684e9f3" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 33829 } } peers { permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 } }
I20260808 07:03:03.249745 21315 consensus_queue.cc:260] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 } } peers { permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 } } peers { permanent_uuid: "e37990d179424f26b5dae70488a5c8ef" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 40561 } } peers { permanent_uuid: "740b28ba0e48425a9279f6f51684e9f3" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 33829 } } peers { permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 } }
I20260808 07:03:03.250025 21315 ts_tablet_manager.cc:1428] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 07:03:03.250038 21316 ts_tablet_manager.cc:1428] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260808 07:03:03.250087 20880 heartbeater.cc:503] Master 127.15.250.190:32917 was elected leader, sending a full tablet report...
I20260808 07:03:03.250139 21157 heartbeater.cc:503] Master 127.15.250.190:32917 was elected leader, sending a full tablet report...
I20260808 07:03:03.251384 21317 raft_consensus.cc:348] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 } } peers { permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 } } peers { permanent_uuid: "e37990d179424f26b5dae70488a5c8ef" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 40561 } } peers { permanent_uuid: "740b28ba0e48425a9279f6f51684e9f3" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 33829 } } peers { permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 } }
I20260808 07:03:03.251510 21317 raft_consensus.cc:374] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:03:03.251531 21317 raft_consensus.cc:729] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ede38c8f2ec4f82b36b8f7aa57809bd, State: Initialized, Role: FOLLOWER
I20260808 07:03:03.251660 21317 consensus_queue.cc:260] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 } } peers { permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 } } peers { permanent_uuid: "e37990d179424f26b5dae70488a5c8ef" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 40561 } } peers { permanent_uuid: "740b28ba0e48425a9279f6f51684e9f3" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 33829 } } peers { permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 } }
I20260808 07:03:03.251962 21317 ts_tablet_manager.cc:1428] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 07:03:03.251999 20741 heartbeater.cc:503] Master 127.15.250.190:32917 was elected leader, sending a full tablet report...
I20260808 07:03:03.252599 21319 tablet_bootstrap.cc:492] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3: No bootstrap required, opened a new log
I20260808 07:03:03.252686 21319 ts_tablet_manager.cc:1397] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260808 07:03:03.254560 21319 raft_consensus.cc:348] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 } } peers { permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 } } peers { permanent_uuid: "e37990d179424f26b5dae70488a5c8ef" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 40561 } } peers { permanent_uuid: "740b28ba0e48425a9279f6f51684e9f3" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 33829 } } peers { permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 } }
I20260808 07:03:03.254724 21319 raft_consensus.cc:374] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:03:03.254770 21319 raft_consensus.cc:729] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 740b28ba0e48425a9279f6f51684e9f3, State: Initialized, Role: FOLLOWER
I20260808 07:03:03.254879 21319 consensus_queue.cc:260] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 } } peers { permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 } } peers { permanent_uuid: "e37990d179424f26b5dae70488a5c8ef" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 40561 } } peers { permanent_uuid: "740b28ba0e48425a9279f6f51684e9f3" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 33829 } } peers { permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 } }
I20260808 07:03:03.255144 21319 ts_tablet_manager.cc:1428] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260808 07:03:03.255203 21292 heartbeater.cc:503] Master 127.15.250.190:32917 was elected leader, sending a full tablet report...
W20260808 07:03:03.317552 20881 tablet.cc:2369] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 07:03:03.321635 21158 tablet.cc:2369] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 07:03:03.426697 21328 raft_consensus.cc:482] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 07:03:03.426869 21328 raft_consensus.cc:504] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 } } peers { permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 } } peers { permanent_uuid: "e37990d179424f26b5dae70488a5c8ef" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 40561 } } peers { permanent_uuid: "740b28ba0e48425a9279f6f51684e9f3" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 33829 } } peers { permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 } }
I20260808 07:03:03.427364 21328 leader_election.cc:288] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a0f4ebf6ab9468898d8d6f756cc81f4 (127.15.250.132:46195), da18746c3d9d4ad0844c8f3c9197478f (127.15.250.130:37825), e37990d179424f26b5dae70488a5c8ef (127.15.250.131:40561), 4ede38c8f2ec4f82b36b8f7aa57809bd (127.15.250.129:34097)
I20260808 07:03:03.431507 20833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "740b28ba0e48425a9279f6f51684e9f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da18746c3d9d4ad0844c8f3c9197478f" is_pre_election: true
I20260808 07:03:03.431706 20833 raft_consensus.cc:2454] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 740b28ba0e48425a9279f6f51684e9f3 in term 0.
I20260808 07:03:03.431948 20969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "740b28ba0e48425a9279f6f51684e9f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e37990d179424f26b5dae70488a5c8ef" is_pre_election: true
I20260808 07:03:03.432099 20969 raft_consensus.cc:2454] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 740b28ba0e48425a9279f6f51684e9f3 in term 0.
I20260808 07:03:03.432072 21107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "740b28ba0e48425a9279f6f51684e9f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" is_pre_election: true
I20260808 07:03:03.432228 21107 raft_consensus.cc:2454] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 740b28ba0e48425a9279f6f51684e9f3 in term 0.
I20260808 07:03:03.432315 21177 leader_election.cc:302] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 740b28ba0e48425a9279f6f51684e9f3, da18746c3d9d4ad0844c8f3c9197478f, e37990d179424f26b5dae70488a5c8ef; no voters:
I20260808 07:03:03.432461 21328 raft_consensus.cc:2781] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 07:03:03.432525 21328 raft_consensus.cc:482] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 07:03:03.432564 21328 raft_consensus.cc:3037] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:03.433028 20691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "740b28ba0e48425a9279f6f51684e9f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" is_pre_election: true
I20260808 07:03:03.433195 20691 raft_consensus.cc:2454] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 740b28ba0e48425a9279f6f51684e9f3 in term 0.
I20260808 07:03:03.433506 21328 raft_consensus.cc:504] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 } } peers { permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 } } peers { permanent_uuid: "e37990d179424f26b5dae70488a5c8ef" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 40561 } } peers { permanent_uuid: "740b28ba0e48425a9279f6f51684e9f3" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 33829 } } peers { permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 } }
I20260808 07:03:03.433741 21328 leader_election.cc:288] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3 [CANDIDATE]: Term 1 election: Requested vote from peers 3a0f4ebf6ab9468898d8d6f756cc81f4 (127.15.250.132:46195), da18746c3d9d4ad0844c8f3c9197478f (127.15.250.130:37825), e37990d179424f26b5dae70488a5c8ef (127.15.250.131:40561), 4ede38c8f2ec4f82b36b8f7aa57809bd (127.15.250.129:34097)
I20260808 07:03:03.433806 20969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "740b28ba0e48425a9279f6f51684e9f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e37990d179424f26b5dae70488a5c8ef"
I20260808 07:03:03.433889 21107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "740b28ba0e48425a9279f6f51684e9f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4"
I20260808 07:03:03.433890 20969 raft_consensus.cc:3037] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:03.433938 20691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "740b28ba0e48425a9279f6f51684e9f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd"
I20260808 07:03:03.433975 21107 raft_consensus.cc:3037] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:03.434005 20691 raft_consensus.cc:3037] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:03.434552 20969 raft_consensus.cc:2454] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 740b28ba0e48425a9279f6f51684e9f3 in term 1.
I20260808 07:03:03.434922 21107 raft_consensus.cc:2454] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 740b28ba0e48425a9279f6f51684e9f3 in term 1.
I20260808 07:03:03.434922 20691 raft_consensus.cc:2454] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 740b28ba0e48425a9279f6f51684e9f3 in term 1.
I20260808 07:03:03.435097 21176 leader_election.cc:302] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3a0f4ebf6ab9468898d8d6f756cc81f4, 740b28ba0e48425a9279f6f51684e9f3, e37990d179424f26b5dae70488a5c8ef; no voters:
I20260808 07:03:03.435088 20833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "740b28ba0e48425a9279f6f51684e9f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da18746c3d9d4ad0844c8f3c9197478f"
I20260808 07:03:03.435190 21328 raft_consensus.cc:2781] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:03.435171 20833 raft_consensus.cc:3037] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:03.435324 21328 raft_consensus.cc:686] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3 [term 1 LEADER]: Becoming Leader. State: Replica: 740b28ba0e48425a9279f6f51684e9f3, State: Running, Role: LEADER
I20260808 07:03:03.435391 21328 consensus_queue.cc:237] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 } } peers { permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 } } peers { permanent_uuid: "e37990d179424f26b5dae70488a5c8ef" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 40561 } } peers { permanent_uuid: "740b28ba0e48425a9279f6f51684e9f3" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 33829 } } peers { permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 } }
I20260808 07:03:03.436053 20833 raft_consensus.cc:2454] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 740b28ba0e48425a9279f6f51684e9f3 in term 1.
W20260808 07:03:03.436225 20742 tablet.cc:2369] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 07:03:03.436786 20542 catalog_manager.cc:5495] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 740b28ba0e48425a9279f6f51684e9f3 (127.15.250.133). New cstate: current_term: 1 leader_uuid: "740b28ba0e48425a9279f6f51684e9f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e37990d179424f26b5dae70488a5c8ef" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 40561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "740b28ba0e48425a9279f6f51684e9f3" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 33829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 } health_report { overall_health: UNKNOWN } } }
W20260808 07:03:03.446624 21019 tablet.cc:2369] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 07:03:03.458137 21293 tablet.cc:2369] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 07:03:03.464069 16362 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20260808 07:03:03.464671 16362 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4ede38c8f2ec4f82b36b8f7aa57809bd to finish bootstrapping
I20260808 07:03:03.468639 16362 ts_itest-base.cc:233] Waiting for 1 tablets on tserver da18746c3d9d4ad0844c8f3c9197478f to finish bootstrapping
I20260808 07:03:03.471971 16362 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e37990d179424f26b5dae70488a5c8ef to finish bootstrapping
I20260808 07:03:03.475665 16362 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 3a0f4ebf6ab9468898d8d6f756cc81f4 to finish bootstrapping
I20260808 07:03:03.479242 16362 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 740b28ba0e48425a9279f6f51684e9f3 to finish bootstrapping
I20260808 07:03:03.486901 16362 raft_consensus_election-itest.cc:397] Writing data to leader of 5-node config (5 alive)...
I20260808 07:03:03.492913 20691 raft_consensus.cc:1261] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [term 1 FOLLOWER]: Refusing update from remote peer 740b28ba0e48425a9279f6f51684e9f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 07:03:03.492913 21107 raft_consensus.cc:1261] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [term 1 FOLLOWER]: Refusing update from remote peer 740b28ba0e48425a9279f6f51684e9f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 07:03:03.492960 20969 raft_consensus.cc:1261] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef [term 1 FOLLOWER]: Refusing update from remote peer 740b28ba0e48425a9279f6f51684e9f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 07:03:03.493290 20833 raft_consensus.cc:1261] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f [term 1 FOLLOWER]: Refusing update from remote peer 740b28ba0e48425a9279f6f51684e9f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 07:03:03.493295 21339 consensus_queue.cc:1035] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:03:03.493422 21338 consensus_queue.cc:1035] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e37990d179424f26b5dae70488a5c8ef" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:03:03.493801 21332 consensus_queue.cc:1035] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:03:03.494280 21328 consensus_queue.cc:1035] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:03:03.496995 21341 mvcc.cc:205] Tried to move back new op lower bound from 7316162901983481856 to 7316162901752094720. Current Snapshot: MvccSnapshot[applied={T|T < 7316162901983481856}]
I20260808 07:03:03.497195 21340 mvcc.cc:205] Tried to move back new op lower bound from 7316162901983481856 to 7316162901752094720. Current Snapshot: MvccSnapshot[applied={T|T < 7316162901983481856}]
I20260808 07:03:03.499079 21344 mvcc.cc:205] Tried to move back new op lower bound from 7316162901983481856 to 7316162901752094720. Current Snapshot: MvccSnapshot[applied={T|T < 7316162901983481856}]
I20260808 07:03:03.499917 21346 mvcc.cc:205] Tried to move back new op lower bound from 7316162901983481856 to 7316162901752094720. Current Snapshot: MvccSnapshot[applied={T|T < 7316162901983481856 or (T in {7316162901983481856})}]
I20260808 07:03:03.506580 16362 raft_consensus_election-itest.cc:408] Killing current leader 740b28ba0e48425a9279f6f51684e9f3...
I20260808 07:03:03.506739 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 21160
W20260808 07:03:03.513360 21039 connection.cc:535] server connection from 127.15.250.133:33247 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 07:03:03.513859 16362 raft_consensus_election-itest.cc:412] Waiting for new guy to be elected leader.
I20260808 07:03:03.514846 16362 test_util.cc:268] Using random seed: -1235676454
I20260808 07:03:03.629575 16362 test_util.cc:268] Using random seed: -1235561731
I20260808 07:03:03.832581 16362 test_util.cc:268] Using random seed: -1235358724
I20260808 07:03:04.135877 16362 test_util.cc:268] Using random seed: -1235055430
I20260808 07:03:04.539270 16362 test_util.cc:268] Using random seed: -1234652039
I20260808 07:03:05.006868 21353 raft_consensus.cc:482] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 740b28ba0e48425a9279f6f51684e9f3)
I20260808 07:03:05.006970 21353 raft_consensus.cc:504] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 } } peers { permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 } } peers { permanent_uuid: "e37990d179424f26b5dae70488a5c8ef" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 40561 } } peers { permanent_uuid: "740b28ba0e48425a9279f6f51684e9f3" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 33829 } } peers { permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 } }
I20260808 07:03:05.007537 21353 leader_election.cc:288] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers da18746c3d9d4ad0844c8f3c9197478f (127.15.250.130:37825), e37990d179424f26b5dae70488a5c8ef (127.15.250.131:40561), 740b28ba0e48425a9279f6f51684e9f3 (127.15.250.133:33829), 4ede38c8f2ec4f82b36b8f7aa57809bd (127.15.250.129:34097)
I20260808 07:03:05.007390 21354 raft_consensus.cc:482] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 740b28ba0e48425a9279f6f51684e9f3)
I20260808 07:03:05.007858 21355 raft_consensus.cc:482] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 740b28ba0e48425a9279f6f51684e9f3)
I20260808 07:03:05.007829 21354 raft_consensus.cc:504] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 } } peers { permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 } } peers { permanent_uuid: "e37990d179424f26b5dae70488a5c8ef" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 40561 } } peers { permanent_uuid: "740b28ba0e48425a9279f6f51684e9f3" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 33829 } } peers { permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 } }
I20260808 07:03:05.007992 21355 raft_consensus.cc:504] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 } } peers { permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 } } peers { permanent_uuid: "e37990d179424f26b5dae70488a5c8ef" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 40561 } } peers { permanent_uuid: "740b28ba0e48425a9279f6f51684e9f3" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 33829 } } peers { permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 } }
I20260808 07:03:05.008167 21363 raft_consensus.cc:482] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 740b28ba0e48425a9279f6f51684e9f3)
I20260808 07:03:05.008307 21363 raft_consensus.cc:504] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 } } peers { permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 } } peers { permanent_uuid: "e37990d179424f26b5dae70488a5c8ef" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 40561 } } peers { permanent_uuid: "740b28ba0e48425a9279f6f51684e9f3" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 33829 } } peers { permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 } }
I20260808 07:03:05.009102 21355 leader_election.cc:288] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a0f4ebf6ab9468898d8d6f756cc81f4 (127.15.250.132:46195), da18746c3d9d4ad0844c8f3c9197478f (127.15.250.130:37825), 740b28ba0e48425a9279f6f51684e9f3 (127.15.250.133:33829), 4ede38c8f2ec4f82b36b8f7aa57809bd (127.15.250.129:34097)
I20260808 07:03:05.009974 21363 leader_election.cc:288] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a0f4ebf6ab9468898d8d6f756cc81f4 (127.15.250.132:46195), da18746c3d9d4ad0844c8f3c9197478f (127.15.250.130:37825), e37990d179424f26b5dae70488a5c8ef (127.15.250.131:40561), 740b28ba0e48425a9279f6f51684e9f3 (127.15.250.133:33829)
I20260808 07:03:05.010077 21354 leader_election.cc:288] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a0f4ebf6ab9468898d8d6f756cc81f4 (127.15.250.132:46195), e37990d179424f26b5dae70488a5c8ef (127.15.250.131:40561), 740b28ba0e48425a9279f6f51684e9f3 (127.15.250.133:33829), 4ede38c8f2ec4f82b36b8f7aa57809bd (127.15.250.129:34097)
W20260808 07:03:05.010572 20904 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.250.133:33829: connect: Connection refused (error 111)
W20260808 07:03:05.012081 20764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.250.133:33829: connect: Connection refused (error 111)
W20260808 07:03:05.012984 20624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.250.133:33829: connect: Connection refused (error 111)
I20260808 07:03:05.013715 20833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "e37990d179424f26b5dae70488a5c8ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "da18746c3d9d4ad0844c8f3c9197478f" is_pre_election: true
I20260808 07:03:05.013827 20833 raft_consensus.cc:2454] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e37990d179424f26b5dae70488a5c8ef in term 1.
W20260808 07:03:05.014922 21043 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.250.133:33829: connect: Connection refused (error 111)
I20260808 07:03:05.015179 20691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" is_pre_election: true
I20260808 07:03:05.015272 20691 raft_consensus.cc:2454] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a0f4ebf6ab9468898d8d6f756cc81f4 in term 1.
W20260808 07:03:05.017262 20764 leader_election.cc:334] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 740b28ba0e48425a9279f6f51684e9f3 (127.15.250.133:33829): Network error: Client connection negotiation failed: client connection to 127.15.250.133:33829: connect: Connection refused (error 111)
I20260808 07:03:05.017351 21107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "e37990d179424f26b5dae70488a5c8ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" is_pre_election: true
I20260808 07:03:05.017438 21107 raft_consensus.cc:2454] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e37990d179424f26b5dae70488a5c8ef in term 1.
I20260808 07:03:05.017665 20901 leader_election.cc:302] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3a0f4ebf6ab9468898d8d6f756cc81f4, da18746c3d9d4ad0844c8f3c9197478f, e37990d179424f26b5dae70488a5c8ef; no voters:
I20260808 07:03:05.017791 21355 raft_consensus.cc:2781] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 07:03:05.017818 21355 raft_consensus.cc:482] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef [term 1 FOLLOWER]: Starting leader election (detected failure of leader 740b28ba0e48425a9279f6f51684e9f3)
I20260808 07:03:05.017838 21355 raft_consensus.cc:3037] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef [term 1 FOLLOWER]: Advancing to term 2
I20260808 07:03:05.017885 20833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "da18746c3d9d4ad0844c8f3c9197478f" is_pre_election: true
I20260808 07:03:05.017966 20833 raft_consensus.cc:2454] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a0f4ebf6ab9468898d8d6f756cc81f4 in term 1.
I20260808 07:03:05.018106 21043 leader_election.cc:302] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3a0f4ebf6ab9468898d8d6f756cc81f4, 4ede38c8f2ec4f82b36b8f7aa57809bd, da18746c3d9d4ad0844c8f3c9197478f; no voters:
I20260808 07:03:05.018220 21353 raft_consensus.cc:2781] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 07:03:05.018254 21353 raft_consensus.cc:482] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 740b28ba0e48425a9279f6f51684e9f3)
I20260808 07:03:05.018280 21353 raft_consensus.cc:3037] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 07:03:05.018621 20691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "e37990d179424f26b5dae70488a5c8ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" is_pre_election: true
I20260808 07:03:05.018699 20691 raft_consensus.cc:2454] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e37990d179424f26b5dae70488a5c8ef in term 1.
I20260808 07:03:05.018836 21355 raft_consensus.cc:504] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 } } peers { permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 } } peers { permanent_uuid: "e37990d179424f26b5dae70488a5c8ef" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 40561 } } peers { permanent_uuid: "740b28ba0e48425a9279f6f51684e9f3" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 33829 } } peers { permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 } }
I20260808 07:03:05.018987 21355 leader_election.cc:288] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef [CANDIDATE]: Term 2 election: Requested vote from peers 3a0f4ebf6ab9468898d8d6f756cc81f4 (127.15.250.132:46195), da18746c3d9d4ad0844c8f3c9197478f (127.15.250.130:37825), 740b28ba0e48425a9279f6f51684e9f3 (127.15.250.133:33829), 4ede38c8f2ec4f82b36b8f7aa57809bd (127.15.250.129:34097)
W20260808 07:03:05.019151 20904 leader_election.cc:334] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 740b28ba0e48425a9279f6f51684e9f3 (127.15.250.133:33829): Network error: Client connection negotiation failed: client connection to 127.15.250.133:33829: connect: Connection refused (error 111)
I20260808 07:03:05.019163 20691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "e37990d179424f26b5dae70488a5c8ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd"
I20260808 07:03:05.019158 21353 raft_consensus.cc:504] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 } } peers { permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 } } peers { permanent_uuid: "e37990d179424f26b5dae70488a5c8ef" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 40561 } } peers { permanent_uuid: "740b28ba0e48425a9279f6f51684e9f3" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 33829 } } peers { permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 } }
I20260808 07:03:05.019220 20691 raft_consensus.cc:3037] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [term 1 FOLLOWER]: Advancing to term 2
I20260808 07:03:05.019291 21353 leader_election.cc:288] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [CANDIDATE]: Term 2 election: Requested vote from peers da18746c3d9d4ad0844c8f3c9197478f (127.15.250.130:37825), e37990d179424f26b5dae70488a5c8ef (127.15.250.131:40561), 740b28ba0e48425a9279f6f51684e9f3 (127.15.250.133:33829), 4ede38c8f2ec4f82b36b8f7aa57809bd (127.15.250.129:34097)
I20260808 07:03:05.019523 20833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "e37990d179424f26b5dae70488a5c8ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "da18746c3d9d4ad0844c8f3c9197478f"
I20260808 07:03:05.019590 20833 raft_consensus.cc:3037] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f [term 1 FOLLOWER]: Advancing to term 2
I20260808 07:03:05.020061 20691 raft_consensus.cc:2454] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e37990d179424f26b5dae70488a5c8ef in term 2.
I20260808 07:03:05.020551 21107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "e37990d179424f26b5dae70488a5c8ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4"
I20260808 07:03:05.020628 21107 raft_consensus.cc:2379] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e37990d179424f26b5dae70488a5c8ef in current term 2: Already voted for candidate 3a0f4ebf6ab9468898d8d6f756cc81f4 in this term.
I20260808 07:03:05.020849 20833 raft_consensus.cc:2454] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e37990d179424f26b5dae70488a5c8ef in term 2.
W20260808 07:03:05.020855 21043 leader_election.cc:334] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 740b28ba0e48425a9279f6f51684e9f3 (127.15.250.133:33829): Network error: Client connection negotiation failed: client connection to 127.15.250.133:33829: connect: Connection refused (error 111)
I20260808 07:03:05.020924 20691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd"
I20260808 07:03:05.020962 20832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "da18746c3d9d4ad0844c8f3c9197478f"
I20260808 07:03:05.020999 20691 raft_consensus.cc:2379] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a0f4ebf6ab9468898d8d6f756cc81f4 in current term 2: Already voted for candidate e37990d179424f26b5dae70488a5c8ef in this term.
I20260808 07:03:05.021037 20832 raft_consensus.cc:2379] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a0f4ebf6ab9468898d8d6f756cc81f4 in current term 2: Already voted for candidate e37990d179424f26b5dae70488a5c8ef in this term.
W20260808 07:03:05.021291 20624 leader_election.cc:334] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 740b28ba0e48425a9279f6f51684e9f3 (127.15.250.133:33829): Network error: Client connection negotiation failed: client connection to 127.15.250.133:33829: connect: Connection refused (error 111)
I20260808 07:03:05.021417 20904 leader_election.cc:302] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 4ede38c8f2ec4f82b36b8f7aa57809bd, da18746c3d9d4ad0844c8f3c9197478f, e37990d179424f26b5dae70488a5c8ef; no voters: 3a0f4ebf6ab9468898d8d6f756cc81f4
I20260808 07:03:05.021499 21355 raft_consensus.cc:2781] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef [term 2 FOLLOWER]: Leader election won for term 2
I20260808 07:03:05.021636 21355 raft_consensus.cc:686] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef [term 2 LEADER]: Becoming Leader. State: Replica: e37990d179424f26b5dae70488a5c8ef, State: Running, Role: LEADER
I20260808 07:03:05.021723 21355 consensus_queue.cc:237] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 } } peers { permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 } } peers { permanent_uuid: "e37990d179424f26b5dae70488a5c8ef" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 40561 } } peers { permanent_uuid: "740b28ba0e48425a9279f6f51684e9f3" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 33829 } } peers { permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 } }
W20260808 07:03:05.021970 21043 leader_election.cc:334] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 740b28ba0e48425a9279f6f51684e9f3 (127.15.250.133:33829): Network error: Client connection negotiation failed: client connection to 127.15.250.133:33829: connect: Connection refused (error 111)
I20260808 07:03:05.022014 21043 leader_election.cc:302] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 3a0f4ebf6ab9468898d8d6f756cc81f4; no voters: 4ede38c8f2ec4f82b36b8f7aa57809bd, 740b28ba0e48425a9279f6f51684e9f3, da18746c3d9d4ad0844c8f3c9197478f
I20260808 07:03:05.022087 21353 raft_consensus.cc:2726] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 07:03:05.022475 20691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "da18746c3d9d4ad0844c8f3c9197478f" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" is_pre_election: true
I20260808 07:03:05.022540 20691 raft_consensus.cc:2379] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate da18746c3d9d4ad0844c8f3c9197478f in current term 2: Already voted for candidate e37990d179424f26b5dae70488a5c8ef in this term.
I20260808 07:03:05.022529 20537 catalog_manager.cc:5495] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef reported cstate change: term changed from 1 to 2, leader changed from 740b28ba0e48425a9279f6f51684e9f3 (127.15.250.133) to e37990d179424f26b5dae70488a5c8ef (127.15.250.131). New cstate: current_term: 2 leader_uuid: "e37990d179424f26b5dae70488a5c8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e37990d179424f26b5dae70488a5c8ef" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 40561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "740b28ba0e48425a9279f6f51684e9f3" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 33829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 } health_report { overall_health: UNKNOWN } } }
I20260808 07:03:05.022821 20969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "e37990d179424f26b5dae70488a5c8ef" is_pre_election: true
W20260808 07:03:05.023008 20904 leader_election.cc:334] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 740b28ba0e48425a9279f6f51684e9f3 (127.15.250.133:33829): Network error: Client connection negotiation failed: client connection to 127.15.250.133:33829: connect: Connection refused (error 111)
I20260808 07:03:05.023010 20968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "e37990d179424f26b5dae70488a5c8ef"
I20260808 07:03:05.023509 21107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" is_pre_election: true
I20260808 07:03:05.023595 21107 raft_consensus.cc:2379] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4ede38c8f2ec4f82b36b8f7aa57809bd in current term 2: Already voted for candidate 3a0f4ebf6ab9468898d8d6f756cc81f4 in this term.
I20260808 07:03:05.023900 20832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "da18746c3d9d4ad0844c8f3c9197478f" is_pre_election: true
I20260808 07:03:05.023973 20832 raft_consensus.cc:2379] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4ede38c8f2ec4f82b36b8f7aa57809bd in current term 2: Already voted for candidate e37990d179424f26b5dae70488a5c8ef in this term.
I20260808 07:03:05.024102 21107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "da18746c3d9d4ad0844c8f3c9197478f" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" is_pre_election: true
I20260808 07:03:05.024169 21107 raft_consensus.cc:2379] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate da18746c3d9d4ad0844c8f3c9197478f in current term 2: Already voted for candidate 3a0f4ebf6ab9468898d8d6f756cc81f4 in this term.
I20260808 07:03:05.024144 20624 leader_election.cc:302] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 4ede38c8f2ec4f82b36b8f7aa57809bd; no voters: 3a0f4ebf6ab9468898d8d6f756cc81f4, 740b28ba0e48425a9279f6f51684e9f3, da18746c3d9d4ad0844c8f3c9197478f
I20260808 07:03:05.024286 21363 raft_consensus.cc:2726] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:03:05.024430 20762 leader_election.cc:302] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: da18746c3d9d4ad0844c8f3c9197478f; no voters: 3a0f4ebf6ab9468898d8d6f756cc81f4, 4ede38c8f2ec4f82b36b8f7aa57809bd, 740b28ba0e48425a9279f6f51684e9f3
I20260808 07:03:05.024598 21354 raft_consensus.cc:2726] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:03:05.025104 20968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "da18746c3d9d4ad0844c8f3c9197478f" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "e37990d179424f26b5dae70488a5c8ef" is_pre_election: true
I20260808 07:03:05.025322 20968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "e37990d179424f26b5dae70488a5c8ef" is_pre_election: true
I20260808 07:03:05.042637 16362 raft_consensus_election-itest.cc:397] Writing data to leader of 5-node config (4 alive)...
W20260808 07:03:05.043708 21298 meta_cache.cc:302] tablet c36a91742d9144c58e84540f45c52a7c: replica 740b28ba0e48425a9279f6f51684e9f3 (127.15.250.133:33829) has failed: Network error: Client connection negotiation failed: client connection to 127.15.250.133:33829: connect: Connection refused (error 111)
W20260808 07:03:05.048403 21067 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47726: Illegal state: replica 3a0f4ebf6ab9468898d8d6f756cc81f4 is not leader of this config: current role FOLLOWER
W20260808 07:03:05.054191 20790 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42964: Illegal state: replica da18746c3d9d4ad0844c8f3c9197478f is not leader of this config: current role FOLLOWER
I20260808 07:03:05.063439 20832 raft_consensus.cc:1261] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f [term 2 FOLLOWER]: Refusing update from remote peer e37990d179424f26b5dae70488a5c8ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20260808 07:03:05.063455 20691 raft_consensus.cc:1261] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [term 2 FOLLOWER]: Refusing update from remote peer e37990d179424f26b5dae70488a5c8ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20260808 07:03:05.063455 21107 raft_consensus.cc:1261] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [term 2 FOLLOWER]: Refusing update from remote peer e37990d179424f26b5dae70488a5c8ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20260808 07:03:05.063681 21402 consensus_queue.cc:1035] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 07:03:05.063858 21391 consensus_queue.cc:1035] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 07:03:05.063936 21355 consensus_queue.cc:1035] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260808 07:03:05.064038 20904 consensus_peers.cc:487] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef -> Peer 740b28ba0e48425a9279f6f51684e9f3 (127.15.250.133:33829): Couldn't send request to peer 740b28ba0e48425a9279f6f51684e9f3. Status: Network error: Client connection negotiation failed: client connection to 127.15.250.133:33829: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 07:03:05.073829 16362 raft_consensus_election-itest.cc:408] Killing current leader e37990d179424f26b5dae70488a5c8ef...
I20260808 07:03:05.073911 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 20883
I20260808 07:03:05.080919 16362 raft_consensus_election-itest.cc:412] Waiting for new guy to be elected leader.
I20260808 07:03:05.081794 16362 test_util.cc:268] Using random seed: -1234109507
I20260808 07:03:05.184718 16362 test_util.cc:268] Using random seed: -1234006595
I20260808 07:03:05.387809 16362 test_util.cc:268] Using random seed: -1233803498
I20260808 07:03:05.691036 16362 test_util.cc:268] Using random seed: -1233500270
I20260808 07:03:06.094805 16362 test_util.cc:268] Using random seed: -1233096502
I20260808 07:03:06.573562 21414 raft_consensus.cc:482] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e37990d179424f26b5dae70488a5c8ef)
I20260808 07:03:06.573671 21414 raft_consensus.cc:504] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 } } peers { permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 } } peers { permanent_uuid: "e37990d179424f26b5dae70488a5c8ef" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 40561 } } peers { permanent_uuid: "740b28ba0e48425a9279f6f51684e9f3" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 33829 } } peers { permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 } }
I20260808 07:03:06.573920 21415 raft_consensus.cc:482] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e37990d179424f26b5dae70488a5c8ef)
I20260808 07:03:06.574008 21414 leader_election.cc:288] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a0f4ebf6ab9468898d8d6f756cc81f4 (127.15.250.132:46195), da18746c3d9d4ad0844c8f3c9197478f (127.15.250.130:37825), e37990d179424f26b5dae70488a5c8ef (127.15.250.131:40561), 740b28ba0e48425a9279f6f51684e9f3 (127.15.250.133:33829)
I20260808 07:03:06.574057 21415 raft_consensus.cc:504] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 } } peers { permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 } } peers { permanent_uuid: "e37990d179424f26b5dae70488a5c8ef" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 40561 } } peers { permanent_uuid: "740b28ba0e48425a9279f6f51684e9f3" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 33829 } } peers { permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 } }
I20260808 07:03:06.574200 21107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" is_pre_election: true
I20260808 07:03:06.574306 21107 raft_consensus.cc:2454] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ede38c8f2ec4f82b36b8f7aa57809bd in term 2.
I20260808 07:03:06.574416 20832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "da18746c3d9d4ad0844c8f3c9197478f" is_pre_election: true
I20260808 07:03:06.574515 21416 raft_consensus.cc:482] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e37990d179424f26b5dae70488a5c8ef)
I20260808 07:03:06.574535 20833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "da18746c3d9d4ad0844c8f3c9197478f" is_pre_election: true
I20260808 07:03:06.574590 21416 raft_consensus.cc:504] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 } } peers { permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 } } peers { permanent_uuid: "e37990d179424f26b5dae70488a5c8ef" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 40561 } } peers { permanent_uuid: "740b28ba0e48425a9279f6f51684e9f3" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 33829 } } peers { permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 } }
I20260808 07:03:06.574849 20832 raft_consensus.cc:2454] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ede38c8f2ec4f82b36b8f7aa57809bd in term 2.
W20260808 07:03:06.575074 20624 leader_election.cc:334] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 740b28ba0e48425a9279f6f51684e9f3 (127.15.250.133:33829): Network error: Client connection negotiation failed: client connection to 127.15.250.133:33829: connect: Connection refused (error 111)
I20260808 07:03:06.575091 21415 leader_election.cc:288] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers da18746c3d9d4ad0844c8f3c9197478f (127.15.250.130:37825), e37990d179424f26b5dae70488a5c8ef (127.15.250.131:40561), 740b28ba0e48425a9279f6f51684e9f3 (127.15.250.133:33829), 4ede38c8f2ec4f82b36b8f7aa57809bd (127.15.250.129:34097)
W20260808 07:03:06.575160 20623 leader_election.cc:334] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e37990d179424f26b5dae70488a5c8ef (127.15.250.131:40561): Network error: Client connection negotiation failed: client connection to 127.15.250.131:40561: connect: Connection refused (error 111)
I20260808 07:03:06.575218 21107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "da18746c3d9d4ad0844c8f3c9197478f" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" is_pre_election: true
I20260808 07:03:06.575287 21107 raft_consensus.cc:2454] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da18746c3d9d4ad0844c8f3c9197478f in term 2.
I20260808 07:03:06.575345 20624 leader_election.cc:302] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 5 voters: 3 yes votes; 2 no votes. yes voters: 3a0f4ebf6ab9468898d8d6f756cc81f4, 4ede38c8f2ec4f82b36b8f7aa57809bd, da18746c3d9d4ad0844c8f3c9197478f; no voters: 740b28ba0e48425a9279f6f51684e9f3, e37990d179424f26b5dae70488a5c8ef
I20260808 07:03:06.575450 21414 raft_consensus.cc:2781] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 07:03:06.575497 21414 raft_consensus.cc:482] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [term 2 FOLLOWER]: Starting leader election (detected failure of leader e37990d179424f26b5dae70488a5c8ef)
I20260808 07:03:06.575503 20691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" is_pre_election: true
I20260808 07:03:06.575536 21414 raft_consensus.cc:3037] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [term 2 FOLLOWER]: Advancing to term 3
W20260808 07:03:06.575759 21043 leader_election.cc:334] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 740b28ba0e48425a9279f6f51684e9f3 (127.15.250.133:33829): Network error: Client connection negotiation failed: client connection to 127.15.250.133:33829: connect: Connection refused (error 111)
W20260808 07:03:06.575837 21042 leader_election.cc:334] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e37990d179424f26b5dae70488a5c8ef (127.15.250.131:40561): Network error: Client connection negotiation failed: client connection to 127.15.250.131:40561: connect: Connection refused (error 111)
I20260808 07:03:06.575870 21042 leader_election.cc:302] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 3a0f4ebf6ab9468898d8d6f756cc81f4; no voters: 740b28ba0e48425a9279f6f51684e9f3, da18746c3d9d4ad0844c8f3c9197478f, e37990d179424f26b5dae70488a5c8ef
I20260808 07:03:06.575954 21415 raft_consensus.cc:2726] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 07:03:06.576190 21416 leader_election.cc:288] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a0f4ebf6ab9468898d8d6f756cc81f4 (127.15.250.132:46195), e37990d179424f26b5dae70488a5c8ef (127.15.250.131:40561), 740b28ba0e48425a9279f6f51684e9f3 (127.15.250.133:33829), 4ede38c8f2ec4f82b36b8f7aa57809bd (127.15.250.129:34097)
I20260808 07:03:06.576212 20689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "da18746c3d9d4ad0844c8f3c9197478f" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" is_pre_election: true
I20260808 07:03:06.576602 21414 raft_consensus.cc:504] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 } } peers { permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 } } peers { permanent_uuid: "e37990d179424f26b5dae70488a5c8ef" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 40561 } } peers { permanent_uuid: "740b28ba0e48425a9279f6f51684e9f3" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 33829 } } peers { permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 } }
I20260808 07:03:06.576759 20691 raft_consensus.cc:2379] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a0f4ebf6ab9468898d8d6f756cc81f4 in current term 3: Already voted for candidate 4ede38c8f2ec4f82b36b8f7aa57809bd in this term.
I20260808 07:03:06.576779 21414 leader_election.cc:288] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [CANDIDATE]: Term 3 election: Requested vote from peers 3a0f4ebf6ab9468898d8d6f756cc81f4 (127.15.250.132:46195), da18746c3d9d4ad0844c8f3c9197478f (127.15.250.130:37825), e37990d179424f26b5dae70488a5c8ef (127.15.250.131:40561), 740b28ba0e48425a9279f6f51684e9f3 (127.15.250.133:33829)
I20260808 07:03:06.576882 21107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4"
I20260808 07:03:06.576946 21107 raft_consensus.cc:3037] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [term 2 FOLLOWER]: Advancing to term 3
W20260808 07:03:06.577248 20763 leader_election.cc:334] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e37990d179424f26b5dae70488a5c8ef (127.15.250.131:40561): Network error: Client connection negotiation failed: client connection to 127.15.250.131:40561: connect: Connection refused (error 111)
W20260808 07:03:06.577356 20623 leader_election.cc:334] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e37990d179424f26b5dae70488a5c8ef (127.15.250.131:40561): Network error: Client connection negotiation failed: client connection to 127.15.250.131:40561: connect: Connection refused (error 111)
W20260808 07:03:06.577543 20764 leader_election.cc:334] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 740b28ba0e48425a9279f6f51684e9f3 (127.15.250.133:33829): Network error: Client connection negotiation failed: client connection to 127.15.250.133:33829: connect: Connection refused (error 111)
I20260808 07:03:06.577705 20764 leader_election.cc:302] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 3a0f4ebf6ab9468898d8d6f756cc81f4, da18746c3d9d4ad0844c8f3c9197478f; no voters: 4ede38c8f2ec4f82b36b8f7aa57809bd, 740b28ba0e48425a9279f6f51684e9f3, e37990d179424f26b5dae70488a5c8ef
I20260808 07:03:06.577796 21107 raft_consensus.cc:2454] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ede38c8f2ec4f82b36b8f7aa57809bd in term 3.
I20260808 07:03:06.577853 20832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a91742d9144c58e84540f45c52a7c" candidate_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "da18746c3d9d4ad0844c8f3c9197478f"
I20260808 07:03:06.577930 20832 raft_consensus.cc:3037] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f [term 2 FOLLOWER]: Advancing to term 3
W20260808 07:03:06.578051 20624 leader_election.cc:334] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 740b28ba0e48425a9279f6f51684e9f3 (127.15.250.133:33829): Network error: Client connection negotiation failed: client connection to 127.15.250.133:33829: connect: Connection refused (error 111)
I20260808 07:03:06.578583 20832 raft_consensus.cc:2454] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ede38c8f2ec4f82b36b8f7aa57809bd in term 3.
I20260808 07:03:06.578691 21416 raft_consensus.cc:2726] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 07:03:06.578825 20624 leader_election.cc:302] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 5 voters: 3 yes votes; 2 no votes. yes voters: 3a0f4ebf6ab9468898d8d6f756cc81f4, 4ede38c8f2ec4f82b36b8f7aa57809bd, da18746c3d9d4ad0844c8f3c9197478f; no voters: 740b28ba0e48425a9279f6f51684e9f3, e37990d179424f26b5dae70488a5c8ef
I20260808 07:03:06.578996 21414 raft_consensus.cc:2781] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [term 3 FOLLOWER]: Leader election won for term 3
I20260808 07:03:06.579183 21414 raft_consensus.cc:686] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [term 3 LEADER]: Becoming Leader. State: Replica: 4ede38c8f2ec4f82b36b8f7aa57809bd, State: Running, Role: LEADER
I20260808 07:03:06.579303 21414 consensus_queue.cc:237] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 } } peers { permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 } } peers { permanent_uuid: "e37990d179424f26b5dae70488a5c8ef" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 40561 } } peers { permanent_uuid: "740b28ba0e48425a9279f6f51684e9f3" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 33829 } } peers { permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 } }
I20260808 07:03:06.580089 20538 catalog_manager.cc:5495] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd reported cstate change: term changed from 2 to 3, leader changed from e37990d179424f26b5dae70488a5c8ef (127.15.250.131) to 4ede38c8f2ec4f82b36b8f7aa57809bd (127.15.250.129). New cstate: current_term: 3 leader_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e37990d179424f26b5dae70488a5c8ef" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 40561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "740b28ba0e48425a9279f6f51684e9f3" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 33829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 } health_report { overall_health: HEALTHY } } }
I20260808 07:03:06.597961 16362 raft_consensus_election-itest.cc:397] Writing data to leader of 5-node config (3 alive)...
W20260808 07:03:06.599182 21297 meta_cache.cc:302] tablet c36a91742d9144c58e84540f45c52a7c: replica e37990d179424f26b5dae70488a5c8ef (127.15.250.131:40561) has failed: Network error: Client connection negotiation failed: client connection to 127.15.250.131:40561: connect: Connection refused (error 111)
W20260808 07:03:06.600660 21067 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47726: Illegal state: replica 3a0f4ebf6ab9468898d8d6f756cc81f4 is not leader of this config: current role FOLLOWER
W20260808 07:03:06.603464 20790 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42964: Illegal state: replica da18746c3d9d4ad0844c8f3c9197478f is not leader of this config: current role FOLLOWER
I20260808 07:03:06.611536 21107 raft_consensus.cc:1261] T c36a91742d9144c58e84540f45c52a7c P 3a0f4ebf6ab9468898d8d6f756cc81f4 [term 3 FOLLOWER]: Refusing update from remote peer 4ede38c8f2ec4f82b36b8f7aa57809bd: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260808 07:03:06.611631 20832 raft_consensus.cc:1261] T c36a91742d9144c58e84540f45c52a7c P da18746c3d9d4ad0844c8f3c9197478f [term 3 FOLLOWER]: Refusing update from remote peer 4ede38c8f2ec4f82b36b8f7aa57809bd: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
W20260808 07:03:06.611717 20624 consensus_peers.cc:487] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd -> Peer 740b28ba0e48425a9279f6f51684e9f3 (127.15.250.133:33829): Couldn't send request to peer 740b28ba0e48425a9279f6f51684e9f3. Status: Network error: Client connection negotiation failed: client connection to 127.15.250.133:33829: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260808 07:03:06.611727 20623 consensus_peers.cc:487] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd -> Peer e37990d179424f26b5dae70488a5c8ef (127.15.250.131:40561): Couldn't send request to peer e37990d179424f26b5dae70488a5c8ef. Status: Network error: Client connection negotiation failed: client connection to 127.15.250.131:40561: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 07:03:06.611910 21437 consensus_queue.cc:1035] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260808 07:03:06.612056 21427 consensus_queue.cc:1035] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260808 07:03:06.619645 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.131:40561
--local_ip_for_outbound_sockets=127.15.250.131
--tserver_master_addrs=127.15.250.190:32917
--webserver_port=42597
--webserver_interface=127.15.250.131
--builtin_ntp_servers=127.15.250.148:45891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
W20260808 07:03:06.705880 21447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:06.706090 21447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:06.706111 21447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:06.707361 21447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:06.707479 21447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.131
I20260808 07:03:06.709005 21447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:45891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.131:40561
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.250.131
--webserver_port=42597
--tserver_master_addrs=127.15.250.190:32917
--never_fsync=true
--heap_profile_path=/tmp/kudu.21447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.131
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:06.709208 21447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:06.709460 21447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:06.714916 21453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:06.714931 21455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:06.714964 21447 server_base.cc:1034] running on GCE node
W20260808 07:03:06.714931 21452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:06.715584 21447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:06.715866 21447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:06.717075 21447 hybrid_clock.cc:648] HybridClock initialized: now 1786172586717059 us; error 45 us; skew 500 ppm
I20260808 07:03:06.718488 21447 webserver.cc:460] Webserver started at http://127.15.250.131:42597/ using document root <none> and password file <none>
I20260808 07:03:06.718771 21447 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:06.718863 21447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:06.720356 21447 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:03:06.721421 21461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:06.721607 21447 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:03:06.721731 21447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e37990d179424f26b5dae70488a5c8ef"
format_stamp: "Formatted at 2026-08-08 07:03:02 on dist-test-slave-0b58"
I20260808 07:03:06.722158 21447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:06.731005 21447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:06.731333 21447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:06.731479 21447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:06.731714 21447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:03:06.732234 21469 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 07:03:06.733611 21447 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 07:03:06.733688 21447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260808 07:03:06.733734 21447 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 07:03:06.734375 21447 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 07:03:06.734428 21447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260808 07:03:06.734498 21469 tablet_bootstrap.cc:492] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef: Bootstrap starting.
I20260808 07:03:06.741322 21447 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.131:40561
I20260808 07:03:06.741422 21585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.131:40561 every 8 connection(s)
I20260808 07:03:06.741741 21447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260808 07:03:06.743934 21469 log.cc:824] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:06.745942 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 21447
I20260808 07:03:06.746296 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.133:33829
--local_ip_for_outbound_sockets=127.15.250.133
--tserver_master_addrs=127.15.250.190:32917
--webserver_port=36797
--webserver_interface=127.15.250.133
--builtin_ntp_servers=127.15.250.148:45891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20260808 07:03:06.747640 21469 tablet_bootstrap.cc:492] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 07:03:06.747983 21469 tablet_bootstrap.cc:492] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef: Bootstrap complete.
I20260808 07:03:06.747990 21586 heartbeater.cc:347] Connected to a master server at 127.15.250.190:32917
I20260808 07:03:06.748200 21586 heartbeater.cc:464] Registering TS with master...
I20260808 07:03:06.748414 21586 heartbeater.cc:511] Master 127.15.250.190:32917 requested a full tablet report, sending...
I20260808 07:03:06.749104 20539 ts_manager.cc:194] Re-registered known tserver with Master: e37990d179424f26b5dae70488a5c8ef (127.15.250.131:40561)
I20260808 07:03:06.749329 21469 ts_tablet_manager.cc:1397] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.010s
I20260808 07:03:06.749627 20539 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.131:59465
I20260808 07:03:06.750701 21469 raft_consensus.cc:348] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 } } peers { permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 } } peers { permanent_uuid: "e37990d179424f26b5dae70488a5c8ef" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 40561 } } peers { permanent_uuid: "740b28ba0e48425a9279f6f51684e9f3" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 33829 } } peers { permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 } }
I20260808 07:03:06.750939 21469 raft_consensus.cc:729] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e37990d179424f26b5dae70488a5c8ef, State: Initialized, Role: FOLLOWER
I20260808 07:03:06.751076 21469 consensus_queue.cc:260] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 } } peers { permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 } } peers { permanent_uuid: "e37990d179424f26b5dae70488a5c8ef" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 40561 } } peers { permanent_uuid: "740b28ba0e48425a9279f6f51684e9f3" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 33829 } } peers { permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 } }
I20260808 07:03:06.751402 21469 ts_tablet_manager.cc:1428] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260808 07:03:06.751434 21586 heartbeater.cc:503] Master 127.15.250.190:32917 was elected leader, sending a full tablet report...
W20260808 07:03:06.831743 21591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:06.831967 21591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:06.831988 21591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:06.833140 21591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:06.833192 21591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.133
I20260808 07:03:06.834442 21591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:45891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.133:33829
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.15.250.133
--webserver_port=36797
--tserver_master_addrs=127.15.250.190:32917
--never_fsync=true
--heap_profile_path=/tmp/kudu.21591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.133
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:06.834635 21591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:06.834879 21591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:06.840049 21599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:06.840121 21591 server_base.cc:1034] running on GCE node
W20260808 07:03:06.840044 21598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:06.840042 21601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:06.840476 21591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:06.840729 21591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:06.841869 21591 hybrid_clock.cc:648] HybridClock initialized: now 1786172586841859 us; error 31 us; skew 500 ppm
I20260808 07:03:06.843139 21591 webserver.cc:460] Webserver started at http://127.15.250.133:36797/ using document root <none> and password file <none>
I20260808 07:03:06.843400 21591 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:06.843463 21591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:06.845027 21591 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 07:03:06.845819 21607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:06.846055 21591 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 07:03:06.846148 21591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "740b28ba0e48425a9279f6f51684e9f3"
format_stamp: "Formatted at 2026-08-08 07:03:03 on dist-test-slave-0b58"
I20260808 07:03:06.846503 21591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:06.863736 21591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:06.864074 21591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:06.864233 21591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:06.864537 21591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:03:06.865031 21615 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 07:03:06.866448 21591 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 07:03:06.866523 21591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260808 07:03:06.866571 21591 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 07:03:06.867256 21591 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 07:03:06.867314 21591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260808 07:03:06.867421 21615 tablet_bootstrap.cc:492] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3: Bootstrap starting.
I20260808 07:03:06.874961 21591 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.133:33829
I20260808 07:03:06.875120 21729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.133:33829 every 8 connection(s)
I20260808 07:03:06.875428 21591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20260808 07:03:06.876470 21615 log.cc:824] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:06.879439 21615 tablet_bootstrap.cc:492] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=50 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 07:03:06.879797 21615 tablet_bootstrap.cc:492] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3: Bootstrap complete.
I20260808 07:03:06.880954 21615 ts_tablet_manager.cc:1397] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.011s
I20260808 07:03:06.881860 21730 heartbeater.cc:347] Connected to a master server at 127.15.250.190:32917
I20260808 07:03:06.881964 21730 heartbeater.cc:464] Registering TS with master...
I20260808 07:03:06.882149 21730 heartbeater.cc:511] Master 127.15.250.190:32917 requested a full tablet report, sending...
I20260808 07:03:06.882342 21615 raft_consensus.cc:348] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 } } peers { permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 } } peers { permanent_uuid: "e37990d179424f26b5dae70488a5c8ef" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 40561 } } peers { permanent_uuid: "740b28ba0e48425a9279f6f51684e9f3" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 33829 } } peers { permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 } }
I20260808 07:03:06.882674 21615 raft_consensus.cc:729] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 740b28ba0e48425a9279f6f51684e9f3, State: Initialized, Role: FOLLOWER
I20260808 07:03:06.882735 20539 ts_manager.cc:194] Re-registered known tserver with Master: 740b28ba0e48425a9279f6f51684e9f3 (127.15.250.133:33829)
I20260808 07:03:06.882886 21615 consensus_queue.cc:260] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0f4ebf6ab9468898d8d6f756cc81f4" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 46195 } } peers { permanent_uuid: "da18746c3d9d4ad0844c8f3c9197478f" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 37825 } } peers { permanent_uuid: "e37990d179424f26b5dae70488a5c8ef" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 40561 } } peers { permanent_uuid: "740b28ba0e48425a9279f6f51684e9f3" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 33829 } } peers { permanent_uuid: "4ede38c8f2ec4f82b36b8f7aa57809bd" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 34097 } }
I20260808 07:03:06.882937 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 21591
I20260808 07:03:06.883198 21615 ts_tablet_manager.cc:1428] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260808 07:03:06.883208 20539 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.133:49919
I20260808 07:03:06.884073 21730 heartbeater.cc:503] Master 127.15.250.190:32917 was elected leader, sending a full tablet report...
I20260808 07:03:06.911175 20812 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260808 07:03:06.912700 21512 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260808 07:03:06.913952 21661 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260808 07:03:06.917091 21087 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260808 07:03:06.921025 20669 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
4fdbc1f474314637a592745bd0faeee9 | 127.15.250.190:32917 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.250.148:45891 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
3a0f4ebf6ab9468898d8d6f756cc81f4 | 127.15.250.132:46195 | HEALTHY | <none> | 0 | 0
4ede38c8f2ec4f82b36b8f7aa57809bd | 127.15.250.129:34097 | HEALTHY | <none> | 1 | 0
740b28ba0e48425a9279f6f51684e9f3 | 127.15.250.133:33829 | HEALTHY | <none> | 0 | 0
da18746c3d9d4ad0844c8f3c9197478f | 127.15.250.130:37825 | HEALTHY | <none> | 0 | 0
e37990d179424f26b5dae70488a5c8ef | 127.15.250.131:40561 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 5
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 5 server(s) checked
local_ip_for_outbound_sockets | 127.15.250.129 | experimental | 127.15.250.129:34097
local_ip_for_outbound_sockets | 127.15.250.130 | experimental | 127.15.250.130:37825
local_ip_for_outbound_sockets | 127.15.250.131 | experimental | 127.15.250.131:40561
local_ip_for_outbound_sockets | 127.15.250.132 | experimental | 127.15.250.132:46195
local_ip_for_outbound_sockets | 127.15.250.133 | experimental | 127.15.250.133:33829
never_fsync | true | unsafe,advanced | all 5 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 5 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 5 server(s) checked
server_dump_info_format | pb | hidden | all 5 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.15.250.129:34097
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.15.250.130:37825
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.15.250.131:40561
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.15.250.132:46195
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.15.250.133:33829
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.250.148:45891 | all 5 server(s) checked
time_source | builtin | all 5 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 6 server(s) checked
Tablet Summary
Tablet c36a91742d9144c58e84540f45c52a7c of table 'TestTable' is conflicted: 2 replicas' active configs disagree with the leader master's
3a0f4ebf6ab9468898d8d6f756cc81f4 (127.15.250.132:46195): RUNNING
da18746c3d9d4ad0844c8f3c9197478f (127.15.250.130:37825): RUNNING
e37990d179424f26b5dae70488a5c8ef (127.15.250.131:40561): RUNNING
740b28ba0e48425a9279f6f51684e9f3 (127.15.250.133:33829): RUNNING
4ede38c8f2ec4f82b36b8f7aa57809bd (127.15.250.129:34097): RUNNING [LEADER]
All reported replicas are:
A = 3a0f4ebf6ab9468898d8d6f756cc81f4
B = da18746c3d9d4ad0844c8f3c9197478f
C = e37990d179424f26b5dae70488a5c8ef
D = 740b28ba0e48425a9279f6f51684e9f3
E = 4ede38c8f2ec4f82b36b8f7aa57809bd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------------------+--------------+--------------+------------
master | A B C D E* | | | Yes
A | A B C D E* | 3 | -1 | Yes
B | A B C D E* | 3 | -1 | Yes
C | A B C D E | 2 | -1 | Yes
D | A B C D E | 1 | -1 | Yes
E | A B C D E* | 3 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+--------------------+---------------+---------+------------+------------------+-------------
TestTable | 5 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 5
Tables | 1
Tablets | 1
Replicas | 5
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260808 07:03:06.927713 16362 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260808 07:03:07.045687 21533 raft_consensus.cc:3037] T c36a91742d9144c58e84540f45c52a7c P e37990d179424f26b5dae70488a5c8ef [term 2 FOLLOWER]: Advancing to term 3
I20260808 07:03:07.047927 21764 mvcc.cc:205] Tried to move back new op lower bound from 7316162914758447104 to 7316162914629537792. Current Snapshot: MvccSnapshot[applied={T|T < 7316162914758447104}]
I20260808 07:03:07.114238 20669 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260808 07:03:07.115295 21661 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260808 07:03:07.116601 20812 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260808 07:03:07.120527 21087 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260808 07:03:07.120867 21512 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
4fdbc1f474314637a592745bd0faeee9 | 127.15.250.190:32917 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.250.148:45891 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
3a0f4ebf6ab9468898d8d6f756cc81f4 | 127.15.250.132:46195 | HEALTHY | <none> | 0 | 0
4ede38c8f2ec4f82b36b8f7aa57809bd | 127.15.250.129:34097 | HEALTHY | <none> | 1 | 0
740b28ba0e48425a9279f6f51684e9f3 | 127.15.250.133:33829 | HEALTHY | <none> | 0 | 0
da18746c3d9d4ad0844c8f3c9197478f | 127.15.250.130:37825 | HEALTHY | <none> | 0 | 0
e37990d179424f26b5dae70488a5c8ef | 127.15.250.131:40561 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 5
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 5 server(s) checked
local_ip_for_outbound_sockets | 127.15.250.129 | experimental | 127.15.250.129:34097
local_ip_for_outbound_sockets | 127.15.250.130 | experimental | 127.15.250.130:37825
local_ip_for_outbound_sockets | 127.15.250.131 | experimental | 127.15.250.131:40561
local_ip_for_outbound_sockets | 127.15.250.132 | experimental | 127.15.250.132:46195
local_ip_for_outbound_sockets | 127.15.250.133 | experimental | 127.15.250.133:33829
never_fsync | true | unsafe,advanced | all 5 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 5 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 5 server(s) checked
server_dump_info_format | pb | hidden | all 5 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.15.250.129:34097
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.15.250.130:37825
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.15.250.131:40561
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.15.250.132:46195
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.15.250.133:33829
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.250.148:45891 | all 5 server(s) checked
time_source | builtin | all 5 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 6 server(s) checked
Tablet Summary
Tablet c36a91742d9144c58e84540f45c52a7c of table 'TestTable' is conflicted: 1 replicas' active configs disagree with the leader master's
3a0f4ebf6ab9468898d8d6f756cc81f4 (127.15.250.132:46195): RUNNING
da18746c3d9d4ad0844c8f3c9197478f (127.15.250.130:37825): RUNNING
e37990d179424f26b5dae70488a5c8ef (127.15.250.131:40561): RUNNING
740b28ba0e48425a9279f6f51684e9f3 (127.15.250.133:33829): RUNNING
4ede38c8f2ec4f82b36b8f7aa57809bd (127.15.250.129:34097): RUNNING [LEADER]
All reported replicas are:
A = 3a0f4ebf6ab9468898d8d6f756cc81f4
B = da18746c3d9d4ad0844c8f3c9197478f
C = e37990d179424f26b5dae70488a5c8ef
D = 740b28ba0e48425a9279f6f51684e9f3
E = 4ede38c8f2ec4f82b36b8f7aa57809bd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------------------+--------------+--------------+------------
master | A B C D E* | | | Yes
A | A B C D E* | 3 | -1 | Yes
B | A B C D E* | 3 | -1 | Yes
C | A B C D E* | 3 | -1 | Yes
D | A B C D E | 1 | -1 | Yes
E | A B C D E* | 3 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+--------------------+---------------+---------+------------+------------------+-------------
TestTable | 5 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 5
Tables | 1
Tablets | 1
Replicas | 5
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260808 07:03:07.128158 16362 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260808 07:03:07.187674 21682 raft_consensus.cc:3037] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3 [term 1 FOLLOWER]: Advancing to term 3
I20260808 07:03:07.189121 21682 raft_consensus.cc:1261] T c36a91742d9144c58e84540f45c52a7c P 740b28ba0e48425a9279f6f51684e9f3 [term 3 FOLLOWER]: Refusing update from remote peer 4ede38c8f2ec4f82b36b8f7aa57809bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20260808 07:03:07.189509 21446 consensus_queue.cc:1037] T c36a91742d9144c58e84540f45c52a7c P 4ede38c8f2ec4f82b36b8f7aa57809bd [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "740b28ba0e48425a9279f6f51684e9f3" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 33829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 07:03:07.191418 21800 mvcc.cc:205] Tried to move back new op lower bound from 7316162914789277696 to 7316162908249759744. Current Snapshot: MvccSnapshot[applied={T|T < 7316162908417220608}]
I20260808 07:03:07.380650 20669 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260808 07:03:07.381374 21087 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260808 07:03:07.381666 21661 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260808 07:03:07.388020 20812 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260808 07:03:07.389274 21512 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
4fdbc1f474314637a592745bd0faeee9 | 127.15.250.190:32917 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.250.148:45891 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
3a0f4ebf6ab9468898d8d6f756cc81f4 | 127.15.250.132:46195 | HEALTHY | <none> | 0 | 0
4ede38c8f2ec4f82b36b8f7aa57809bd | 127.15.250.129:34097 | HEALTHY | <none> | 1 | 0
740b28ba0e48425a9279f6f51684e9f3 | 127.15.250.133:33829 | HEALTHY | <none> | 0 | 0
da18746c3d9d4ad0844c8f3c9197478f | 127.15.250.130:37825 | HEALTHY | <none> | 0 | 0
e37990d179424f26b5dae70488a5c8ef | 127.15.250.131:40561 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 5
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 5 server(s) checked
local_ip_for_outbound_sockets | 127.15.250.129 | experimental | 127.15.250.129:34097
local_ip_for_outbound_sockets | 127.15.250.130 | experimental | 127.15.250.130:37825
local_ip_for_outbound_sockets | 127.15.250.131 | experimental | 127.15.250.131:40561
local_ip_for_outbound_sockets | 127.15.250.132 | experimental | 127.15.250.132:46195
local_ip_for_outbound_sockets | 127.15.250.133 | experimental | 127.15.250.133:33829
never_fsync | true | unsafe,advanced | all 5 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 5 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 5 server(s) checked
server_dump_info_format | pb | hidden | all 5 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.15.250.129:34097
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.15.250.130:37825
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.15.250.131:40561
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.15.250.132:46195
server_dump_info_path | /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.15.250.133:33829
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.250.148:45891 | all 5 server(s) checked
time_source | builtin | all 5 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 6 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 5 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 5
Tables | 1
Tablets | 1
Replicas | 5
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260808 07:03:07.396009 16362 log_verifier.cc:126] Checking tablet c36a91742d9144c58e84540f45c52a7c
I20260808 07:03:07.427102 16362 log_verifier.cc:177] Verified matching terms for 18 ops in tablet c36a91742d9144c58e84540f45c52a7c
I20260808 07:03:07.437431 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 20605
I20260808 07:03:07.443871 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 20744
I20260808 07:03:07.449757 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 21447
I20260808 07:03:07.455021 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 21022
I20260808 07:03:07.460786 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 21591
I20260808 07:03:07.466684 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 20509
2026-08-08T07:03:07Z chronyd exiting
[ OK ] RaftConsensusElectionITest.AutomaticLeaderElection (5119 ms)
[ RUN ] RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica
I20260808 07:03:07.493381 16362 test_util.cc:268] Using random seed: -1231697921
I20260808 07:03:07.493960 16362 ts_itest-base.cc:112] Starting cluster with:
I20260808 07:03:07.494006 16362 ts_itest-base.cc:113] --------------
I20260808 07:03:07.494019 16362 ts_itest-base.cc:114] 1 tablet servers
I20260808 07:03:07.494029 16362 ts_itest-base.cc:115] 1 replicas per TS
I20260808 07:03:07.494040 16362 ts_itest-base.cc:116] --------------
2026-08-08T07:03:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:03:07Z Disabled control of system clock
I20260808 07:03:07.502882 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.250.190:36561
--webserver_interface=127.15.250.190
--webserver_port=0
--builtin_ntp_servers=127.15.250.148:32785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.250.190:36561 with env {}
W20260808 07:03:07.585028 21850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:07.585299 21850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:07.585335 21850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:07.586511 21850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:03:07.586586 21850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:07.586608 21850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:03:07.586628 21850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:03:07.588024 21850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:32785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.250.190:36561
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.250.190:36561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.250.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.21850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:07.588336 21850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:07.588655 21850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:07.593654 21857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:07.593663 21856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:07.593886 21850 server_base.cc:1034] running on GCE node
W20260808 07:03:07.593663 21860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:07.594245 21850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:07.594507 21850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:07.595649 21850 hybrid_clock.cc:648] HybridClock initialized: now 1786172587595615 us; error 49 us; skew 500 ppm
I20260808 07:03:07.596956 21850 webserver.cc:460] Webserver started at http://127.15.250.190:41469/ using document root <none> and password file <none>
I20260808 07:03:07.597223 21850 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:07.597270 21850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:07.597456 21850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:07.598459 21850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "468716421ad24d68be7672c646494e0a"
format_stamp: "Formatted at 2026-08-08 07:03:07 on dist-test-slave-0b58"
I20260808 07:03:07.598820 21850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "468716421ad24d68be7672c646494e0a"
format_stamp: "Formatted at 2026-08-08 07:03:07 on dist-test-slave-0b58"
I20260808 07:03:07.600157 21850 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 07:03:07.600977 21866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:07.601171 21850 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:03:07.601266 21850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
uuid: "468716421ad24d68be7672c646494e0a"
format_stamp: "Formatted at 2026-08-08 07:03:07 on dist-test-slave-0b58"
I20260808 07:03:07.601344 21850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:07.618400 21850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:07.618763 21850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:07.618934 21850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:07.623298 21850 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.190:36561
I20260808 07:03:07.623358 21920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.190:36561 every 8 connection(s)
I20260808 07:03:07.623750 21850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260808 07:03:07.624604 21921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:07.627570 21921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 468716421ad24d68be7672c646494e0a: Bootstrap starting.
I20260808 07:03:07.627622 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 21850
I20260808 07:03:07.627718 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260808 07:03:07.628459 21921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 468716421ad24d68be7672c646494e0a: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:07.628846 21921 log.cc:824] T 00000000000000000000000000000000 P 468716421ad24d68be7672c646494e0a: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:07.629582 21921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 468716421ad24d68be7672c646494e0a: No bootstrap required, opened a new log
I20260808 07:03:07.631526 21921 raft_consensus.cc:348] T 00000000000000000000000000000000 P 468716421ad24d68be7672c646494e0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "468716421ad24d68be7672c646494e0a" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 36561 } }
I20260808 07:03:07.631676 21921 raft_consensus.cc:374] T 00000000000000000000000000000000 P 468716421ad24d68be7672c646494e0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:03:07.631701 21921 raft_consensus.cc:729] T 00000000000000000000000000000000 P 468716421ad24d68be7672c646494e0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 468716421ad24d68be7672c646494e0a, State: Initialized, Role: FOLLOWER
I20260808 07:03:07.631851 21921 consensus_queue.cc:260] T 00000000000000000000000000000000 P 468716421ad24d68be7672c646494e0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "468716421ad24d68be7672c646494e0a" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 36561 } }
I20260808 07:03:07.631929 21921 raft_consensus.cc:388] T 00000000000000000000000000000000 P 468716421ad24d68be7672c646494e0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:03:07.631953 21921 raft_consensus.cc:482] T 00000000000000000000000000000000 P 468716421ad24d68be7672c646494e0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:03:07.632021 21921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 468716421ad24d68be7672c646494e0a [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:07.633200 21921 raft_consensus.cc:504] T 00000000000000000000000000000000 P 468716421ad24d68be7672c646494e0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "468716421ad24d68be7672c646494e0a" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 36561 } }
I20260808 07:03:07.633330 21921 leader_election.cc:302] T 00000000000000000000000000000000 P 468716421ad24d68be7672c646494e0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 468716421ad24d68be7672c646494e0a; no voters:
I20260808 07:03:07.633561 21921 leader_election.cc:288] T 00000000000000000000000000000000 P 468716421ad24d68be7672c646494e0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:03:07.633599 21926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 468716421ad24d68be7672c646494e0a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:07.633714 21926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 468716421ad24d68be7672c646494e0a [term 1 LEADER]: Becoming Leader. State: Replica: 468716421ad24d68be7672c646494e0a, State: Running, Role: LEADER
I20260808 07:03:07.633821 21926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 468716421ad24d68be7672c646494e0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "468716421ad24d68be7672c646494e0a" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 36561 } }
I20260808 07:03:07.634169 21926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 468716421ad24d68be7672c646494e0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 468716421ad24d68be7672c646494e0a. Latest consensus state: current_term: 1 leader_uuid: "468716421ad24d68be7672c646494e0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "468716421ad24d68be7672c646494e0a" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 36561 } } }
I20260808 07:03:07.634176 21928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 468716421ad24d68be7672c646494e0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "468716421ad24d68be7672c646494e0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "468716421ad24d68be7672c646494e0a" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 36561 } } }
I20260808 07:03:07.634460 21926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 468716421ad24d68be7672c646494e0a [sys.catalog]: This master's current role is: LEADER
I20260808 07:03:07.634476 21928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 468716421ad24d68be7672c646494e0a [sys.catalog]: This master's current role is: LEADER
I20260808 07:03:07.634294 21921 sys_catalog.cc:564] T 00000000000000000000000000000000 P 468716421ad24d68be7672c646494e0a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:03:07.634943 21931 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:03:07.635471 21931 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:03:07.637091 21931 catalog_manager.cc:1347] Generated new cluster ID: c01fce9234884d289a380866fc1dfff7
I20260808 07:03:07.637144 21931 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:03:07.656119 21931 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:03:07.656703 21931 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:03:07.662014 21931 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 468716421ad24d68be7672c646494e0a: Generated new TSK 0
I20260808 07:03:07.662220 21931 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:03:07.665824 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.129:0
--local_ip_for_outbound_sockets=127.15.250.129
--webserver_interface=127.15.250.129
--webserver_port=0
--tserver_master_addrs=127.15.250.190:36561
--builtin_ntp_servers=127.15.250.148:32785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
W20260808 07:03:07.749238 21948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:07.749454 21948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:07.749475 21948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:07.750656 21948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:07.750711 21948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.129
I20260808 07:03:07.752060 21948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:32785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.250.129
--webserver_port=0
--tserver_master_addrs=127.15.250.190:36561
--never_fsync=true
--heap_profile_path=/tmp/kudu.21948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.129
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:07.752350 21948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:07.752681 21948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:07.757856 21953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:07.757838 21954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:07.757951 21948 server_base.cc:1034] running on GCE node
W20260808 07:03:07.757818 21956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:07.758347 21948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:07.758636 21948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:07.759820 21948 hybrid_clock.cc:648] HybridClock initialized: now 1786172587759781 us; error 38 us; skew 500 ppm
I20260808 07:03:07.761312 21948 webserver.cc:460] Webserver started at http://127.15.250.129:44251/ using document root <none> and password file <none>
I20260808 07:03:07.761598 21948 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:07.761672 21948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:07.761837 21948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:07.762862 21948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "ca816bb3eaaf45f5adc90a0353593315"
format_stamp: "Formatted at 2026-08-08 07:03:07 on dist-test-slave-0b58"
I20260808 07:03:07.763229 21948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "ca816bb3eaaf45f5adc90a0353593315"
format_stamp: "Formatted at 2026-08-08 07:03:07 on dist-test-slave-0b58"
I20260808 07:03:07.764735 21948 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 07:03:07.765537 21962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:07.765714 21948 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 07:03:07.765823 21948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "ca816bb3eaaf45f5adc90a0353593315"
format_stamp: "Formatted at 2026-08-08 07:03:07 on dist-test-slave-0b58"
I20260808 07:03:07.765911 21948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:07.774502 21948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:07.774801 21948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:07.774950 21948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:07.775175 21948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:03:07.775501 21948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:03:07.775556 21948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:07.775599 21948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:03:07.775625 21948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:07.781468 21948 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.129:33495
I20260808 07:03:07.781514 22079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.129:33495 every 8 connection(s)
I20260808 07:03:07.781899 21948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260808 07:03:07.787639 22080 heartbeater.cc:347] Connected to a master server at 127.15.250.190:36561
I20260808 07:03:07.787760 22080 heartbeater.cc:464] Registering TS with master...
I20260808 07:03:07.787981 22080 heartbeater.cc:511] Master 127.15.250.190:36561 requested a full tablet report, sending...
I20260808 07:03:07.788623 21884 ts_manager.cc:194] Registered new tserver with Master: ca816bb3eaaf45f5adc90a0353593315 (127.15.250.129:33495)
I20260808 07:03:07.789278 21884 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.129:39015
I20260808 07:03:07.791936 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 21948
I20260808 07:03:07.792016 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260808 07:03:07.793385 16362 external_mini_cluster.cc:930] 1 TS(s) registered with all masters
I20260808 07:03:07.800575 21884 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54074:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20260808 07:03:07.808049 22013 tablet_service.cc:1467] Processing CreateTablet for tablet f5a6e34f0f0341958771b6756e7c635d (DEFAULT_TABLE table=TestTable [id=a5c1def5156441f4b681bcf67ad2dee2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:03:07.808462 22013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5a6e34f0f0341958771b6756e7c635d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:07.811017 22091 tablet_bootstrap.cc:492] T f5a6e34f0f0341958771b6756e7c635d P ca816bb3eaaf45f5adc90a0353593315: Bootstrap starting.
I20260808 07:03:07.811623 22091 tablet_bootstrap.cc:654] T f5a6e34f0f0341958771b6756e7c635d P ca816bb3eaaf45f5adc90a0353593315: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:07.811944 22091 log.cc:824] T f5a6e34f0f0341958771b6756e7c635d P ca816bb3eaaf45f5adc90a0353593315: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:07.812729 22091 tablet_bootstrap.cc:492] T f5a6e34f0f0341958771b6756e7c635d P ca816bb3eaaf45f5adc90a0353593315: No bootstrap required, opened a new log
I20260808 07:03:07.812813 22091 ts_tablet_manager.cc:1397] T f5a6e34f0f0341958771b6756e7c635d P ca816bb3eaaf45f5adc90a0353593315: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:03:07.814623 22091 raft_consensus.cc:348] T f5a6e34f0f0341958771b6756e7c635d P ca816bb3eaaf45f5adc90a0353593315 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca816bb3eaaf45f5adc90a0353593315" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 33495 } }
I20260808 07:03:07.814774 22091 raft_consensus.cc:374] T f5a6e34f0f0341958771b6756e7c635d P ca816bb3eaaf45f5adc90a0353593315 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:03:07.814797 22091 raft_consensus.cc:729] T f5a6e34f0f0341958771b6756e7c635d P ca816bb3eaaf45f5adc90a0353593315 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca816bb3eaaf45f5adc90a0353593315, State: Initialized, Role: FOLLOWER
I20260808 07:03:07.814960 22091 consensus_queue.cc:260] T f5a6e34f0f0341958771b6756e7c635d P ca816bb3eaaf45f5adc90a0353593315 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca816bb3eaaf45f5adc90a0353593315" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 33495 } }
I20260808 07:03:07.815026 22091 raft_consensus.cc:388] T f5a6e34f0f0341958771b6756e7c635d P ca816bb3eaaf45f5adc90a0353593315 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:03:07.815049 22091 raft_consensus.cc:482] T f5a6e34f0f0341958771b6756e7c635d P ca816bb3eaaf45f5adc90a0353593315 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:03:07.815124 22091 raft_consensus.cc:3037] T f5a6e34f0f0341958771b6756e7c635d P ca816bb3eaaf45f5adc90a0353593315 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:07.815761 22091 raft_consensus.cc:504] T f5a6e34f0f0341958771b6756e7c635d P ca816bb3eaaf45f5adc90a0353593315 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca816bb3eaaf45f5adc90a0353593315" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 33495 } }
I20260808 07:03:07.815882 22091 leader_election.cc:302] T f5a6e34f0f0341958771b6756e7c635d P ca816bb3eaaf45f5adc90a0353593315 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca816bb3eaaf45f5adc90a0353593315; no voters:
I20260808 07:03:07.816083 22091 leader_election.cc:288] T f5a6e34f0f0341958771b6756e7c635d P ca816bb3eaaf45f5adc90a0353593315 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:03:07.816111 22093 raft_consensus.cc:2781] T f5a6e34f0f0341958771b6756e7c635d P ca816bb3eaaf45f5adc90a0353593315 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:07.816322 22093 raft_consensus.cc:686] T f5a6e34f0f0341958771b6756e7c635d P ca816bb3eaaf45f5adc90a0353593315 [term 1 LEADER]: Becoming Leader. State: Replica: ca816bb3eaaf45f5adc90a0353593315, State: Running, Role: LEADER
I20260808 07:03:07.816416 22093 consensus_queue.cc:237] T f5a6e34f0f0341958771b6756e7c635d P ca816bb3eaaf45f5adc90a0353593315 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca816bb3eaaf45f5adc90a0353593315" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 33495 } }
I20260808 07:03:07.816437 22091 ts_tablet_manager.cc:1428] T f5a6e34f0f0341958771b6756e7c635d P ca816bb3eaaf45f5adc90a0353593315: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260808 07:03:07.816605 22080 heartbeater.cc:503] Master 127.15.250.190:36561 was elected leader, sending a full tablet report...
I20260808 07:03:07.817394 21884 catalog_manager.cc:5495] T f5a6e34f0f0341958771b6756e7c635d P ca816bb3eaaf45f5adc90a0353593315 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca816bb3eaaf45f5adc90a0353593315 (127.15.250.129). New cstate: current_term: 1 leader_uuid: "ca816bb3eaaf45f5adc90a0353593315" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca816bb3eaaf45f5adc90a0353593315" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 33495 } health_report { overall_health: HEALTHY } } }
I20260808 07:03:07.821988 16362 external_mini_cluster.cc:930] 1 TS(s) registered with all masters
I20260808 07:03:07.822531 16362 ts_itest-base.cc:233] Waiting for 1 tablets on tserver ca816bb3eaaf45f5adc90a0353593315 to finish bootstrapping
I20260808 07:03:07.830032 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 21948
I20260808 07:03:07.837235 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 21850
2026-08-08T07:03:07Z chronyd exiting
[ OK ] RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica (363 ms)
[ RUN ] RaftConsensusElectionITest.LeaderStepDown
I20260808 07:03:07.856793 16362 test_util.cc:268] Using random seed: -1231334508
I20260808 07:03:07.857311 16362 ts_itest-base.cc:112] Starting cluster with:
I20260808 07:03:07.857365 16362 ts_itest-base.cc:113] --------------
I20260808 07:03:07.857383 16362 ts_itest-base.cc:114] 3 tablet servers
I20260808 07:03:07.857393 16362 ts_itest-base.cc:115] 3 replicas per TS
I20260808 07:03:07.857410 16362 ts_itest-base.cc:116] --------------
2026-08-08T07:03:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:03:07Z Disabled control of system clock
I20260808 07:03:07.866480 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.250.190:38697
--webserver_interface=127.15.250.190
--webserver_port=0
--builtin_ntp_servers=127.15.250.148:39149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.250.190:38697
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20260808 07:03:07.949304 22106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:07.949550 22106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:07.949584 22106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:07.950748 22106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:03:07.950816 22106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:07.950838 22106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:03:07.950856 22106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:03:07.952239 22106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:39149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.15.250.190:38697
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.250.190:38697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.250.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.22106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:07.952538 22106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:07.952860 22106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:07.957850 22112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:07.957882 22111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:07.957875 22114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:07.958233 22106 server_base.cc:1034] running on GCE node
I20260808 07:03:07.958492 22106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:07.958819 22106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:07.959997 22106 hybrid_clock.cc:648] HybridClock initialized: now 1786172587959987 us; error 47 us; skew 500 ppm
I20260808 07:03:07.961349 22106 webserver.cc:460] Webserver started at http://127.15.250.190:34305/ using document root <none> and password file <none>
I20260808 07:03:07.961614 22106 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:07.961684 22106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:07.961835 22106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:07.962821 22106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "8f226b1b4dcc45309c40b6fb994cfd3b"
format_stamp: "Formatted at 2026-08-08 07:03:07 on dist-test-slave-0b58"
I20260808 07:03:07.963218 22106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "8f226b1b4dcc45309c40b6fb994cfd3b"
format_stamp: "Formatted at 2026-08-08 07:03:07 on dist-test-slave-0b58"
I20260808 07:03:07.964634 22106 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 07:03:07.965441 22120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:07.965626 22106 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 07:03:07.965734 22106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
uuid: "8f226b1b4dcc45309c40b6fb994cfd3b"
format_stamp: "Formatted at 2026-08-08 07:03:07 on dist-test-slave-0b58"
I20260808 07:03:07.965811 22106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:08.004875 22106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:08.005290 22106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:08.005465 22106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:08.009778 22106 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.190:38697
I20260808 07:03:08.009816 22175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.190:38697 every 8 connection(s)
I20260808 07:03:08.010198 22106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260808 07:03:08.011276 22176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:08.011600 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 22106
I20260808 07:03:08.011746 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260808 07:03:08.014109 22176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f226b1b4dcc45309c40b6fb994cfd3b: Bootstrap starting.
I20260808 07:03:08.014815 22176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f226b1b4dcc45309c40b6fb994cfd3b: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:08.015158 22176 log.cc:824] T 00000000000000000000000000000000 P 8f226b1b4dcc45309c40b6fb994cfd3b: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:08.015864 22176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f226b1b4dcc45309c40b6fb994cfd3b: No bootstrap required, opened a new log
I20260808 07:03:08.017911 22176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f226b1b4dcc45309c40b6fb994cfd3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f226b1b4dcc45309c40b6fb994cfd3b" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 38697 } }
I20260808 07:03:08.018097 22176 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f226b1b4dcc45309c40b6fb994cfd3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:03:08.018167 22176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f226b1b4dcc45309c40b6fb994cfd3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f226b1b4dcc45309c40b6fb994cfd3b, State: Initialized, Role: FOLLOWER
I20260808 07:03:08.018311 22176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f226b1b4dcc45309c40b6fb994cfd3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f226b1b4dcc45309c40b6fb994cfd3b" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 38697 } }
I20260808 07:03:08.018375 22176 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f226b1b4dcc45309c40b6fb994cfd3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:03:08.018397 22176 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f226b1b4dcc45309c40b6fb994cfd3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:03:08.018462 22176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f226b1b4dcc45309c40b6fb994cfd3b [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:08.018975 22176 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f226b1b4dcc45309c40b6fb994cfd3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f226b1b4dcc45309c40b6fb994cfd3b" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 38697 } }
I20260808 07:03:08.019100 22176 leader_election.cc:302] T 00000000000000000000000000000000 P 8f226b1b4dcc45309c40b6fb994cfd3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f226b1b4dcc45309c40b6fb994cfd3b; no voters:
I20260808 07:03:08.019291 22176 leader_election.cc:288] T 00000000000000000000000000000000 P 8f226b1b4dcc45309c40b6fb994cfd3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:03:08.019348 22181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f226b1b4dcc45309c40b6fb994cfd3b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:08.019497 22181 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f226b1b4dcc45309c40b6fb994cfd3b [term 1 LEADER]: Becoming Leader. State: Replica: 8f226b1b4dcc45309c40b6fb994cfd3b, State: Running, Role: LEADER
I20260808 07:03:08.019598 22176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f226b1b4dcc45309c40b6fb994cfd3b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:03:08.019704 22181 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f226b1b4dcc45309c40b6fb994cfd3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f226b1b4dcc45309c40b6fb994cfd3b" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 38697 } }
I20260808 07:03:08.020092 22182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f226b1b4dcc45309c40b6fb994cfd3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f226b1b4dcc45309c40b6fb994cfd3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f226b1b4dcc45309c40b6fb994cfd3b" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 38697 } } }
I20260808 07:03:08.020176 22182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f226b1b4dcc45309c40b6fb994cfd3b [sys.catalog]: This master's current role is: LEADER
I20260808 07:03:08.020380 22183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f226b1b4dcc45309c40b6fb994cfd3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f226b1b4dcc45309c40b6fb994cfd3b. Latest consensus state: current_term: 1 leader_uuid: "8f226b1b4dcc45309c40b6fb994cfd3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f226b1b4dcc45309c40b6fb994cfd3b" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 38697 } } }
I20260808 07:03:08.020431 22190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:03:08.021128 22190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:03:08.020447 22183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f226b1b4dcc45309c40b6fb994cfd3b [sys.catalog]: This master's current role is: LEADER
I20260808 07:03:08.022686 22190 catalog_manager.cc:1347] Generated new cluster ID: 0f3b1969bb394592970a5c314f9f0443
I20260808 07:03:08.022738 22190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:03:08.028434 22190 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:03:08.029094 22190 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:03:08.034948 22190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f226b1b4dcc45309c40b6fb994cfd3b: Generated new TSK 0
I20260808 07:03:08.035142 22190 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:03:08.041150 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.129:0
--local_ip_for_outbound_sockets=127.15.250.129
--webserver_interface=127.15.250.129
--webserver_port=0
--tserver_master_addrs=127.15.250.190:38697
--builtin_ntp_servers=127.15.250.148:39149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20260808 07:03:08.127615 22202 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 07:03:08.127882 22202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:08.127910 22202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:08.127954 22202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:08.129246 22202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:08.129339 22202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.129
I20260808 07:03:08.130772 22202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:39149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.250.129
--webserver_port=0
--tserver_master_addrs=127.15.250.190:38697
--never_fsync=true
--heap_profile_path=/tmp/kudu.22202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.129
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:08.131076 22202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:08.131407 22202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:08.136591 22207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:08.136608 22208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:08.136845 22210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:08.136863 22202 server_base.cc:1034] running on GCE node
I20260808 07:03:08.137063 22202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:08.137336 22202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:08.138521 22202 hybrid_clock.cc:648] HybridClock initialized: now 1786172588138501 us; error 34 us; skew 500 ppm
I20260808 07:03:08.139915 22202 webserver.cc:460] Webserver started at http://127.15.250.129:39987/ using document root <none> and password file <none>
I20260808 07:03:08.140239 22202 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:08.140400 22202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:08.140573 22202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:08.141669 22202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "32dd316e9a274f2183a23289815cb166"
format_stamp: "Formatted at 2026-08-08 07:03:08 on dist-test-slave-0b58"
I20260808 07:03:08.143402 22202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "32dd316e9a274f2183a23289815cb166"
format_stamp: "Formatted at 2026-08-08 07:03:08 on dist-test-slave-0b58"
I20260808 07:03:08.145135 22202 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 07:03:08.146046 22216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:08.146315 22202 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 07:03:08.146433 22202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "32dd316e9a274f2183a23289815cb166"
format_stamp: "Formatted at 2026-08-08 07:03:08 on dist-test-slave-0b58"
I20260808 07:03:08.146518 22202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:08.163745 22202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:08.164050 22202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:08.164157 22202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:08.164431 22202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:03:08.164806 22202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:03:08.164841 22202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:08.164865 22202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:03:08.164915 22202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:08.170885 22202 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.129:44379
I20260808 07:03:08.170936 22333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.129:44379 every 8 connection(s)
I20260808 07:03:08.171314 22202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260808 07:03:08.177477 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 22202
I20260808 07:03:08.177609 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260808 07:03:08.178360 22335 heartbeater.cc:347] Connected to a master server at 127.15.250.190:38697
I20260808 07:03:08.178478 22335 heartbeater.cc:464] Registering TS with master...
I20260808 07:03:08.178699 22335 heartbeater.cc:511] Master 127.15.250.190:38697 requested a full tablet report, sending...
I20260808 07:03:08.179152 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.130:0
--local_ip_for_outbound_sockets=127.15.250.130
--webserver_interface=127.15.250.130
--webserver_port=0
--tserver_master_addrs=127.15.250.190:38697
--builtin_ntp_servers=127.15.250.148:39149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260808 07:03:08.179349 22139 ts_manager.cc:194] Registered new tserver with Master: 32dd316e9a274f2183a23289815cb166 (127.15.250.129:44379)
I20260808 07:03:08.180280 22139 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.129:50587
W20260808 07:03:08.268154 22340 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 07:03:08.268430 22340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:08.268450 22340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:08.268471 22340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:08.269863 22340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:08.269946 22340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.130
I20260808 07:03:08.271322 22340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:39149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.250.130
--webserver_port=0
--tserver_master_addrs=127.15.250.190:38697
--never_fsync=true
--heap_profile_path=/tmp/kudu.22340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.130
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:08.271592 22340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:08.271907 22340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:08.277148 22346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:08.277329 22340 server_base.cc:1034] running on GCE node
W20260808 07:03:08.277199 22345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:08.277200 22348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:08.277724 22340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:08.278007 22340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:08.279201 22340 hybrid_clock.cc:648] HybridClock initialized: now 1786172588279191 us; error 31 us; skew 500 ppm
I20260808 07:03:08.280812 22340 webserver.cc:460] Webserver started at http://127.15.250.130:34571/ using document root <none> and password file <none>
I20260808 07:03:08.281091 22340 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:08.281165 22340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:08.281327 22340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:08.282377 22340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "84283ed4c56d4f92a197f35dfd077790"
format_stamp: "Formatted at 2026-08-08 07:03:08 on dist-test-slave-0b58"
I20260808 07:03:08.282747 22340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "84283ed4c56d4f92a197f35dfd077790"
format_stamp: "Formatted at 2026-08-08 07:03:08 on dist-test-slave-0b58"
I20260808 07:03:08.284220 22340 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 07:03:08.285112 22354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:08.285338 22340 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:03:08.285444 22340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "84283ed4c56d4f92a197f35dfd077790"
format_stamp: "Formatted at 2026-08-08 07:03:08 on dist-test-slave-0b58"
I20260808 07:03:08.285521 22340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:08.305024 22340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:08.305380 22340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:08.305538 22340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:08.305794 22340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:03:08.306167 22340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:03:08.306226 22340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:08.306270 22340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:03:08.306298 22340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:08.312103 22340 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.130:32871
I20260808 07:03:08.312235 22474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.130:32871 every 8 connection(s)
I20260808 07:03:08.312597 22340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260808 07:03:08.315572 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 22340
I20260808 07:03:08.315719 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260808 07:03:08.317257 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.131:0
--local_ip_for_outbound_sockets=127.15.250.131
--webserver_interface=127.15.250.131
--webserver_port=0
--tserver_master_addrs=127.15.250.190:38697
--builtin_ntp_servers=127.15.250.148:39149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260808 07:03:08.319427 22475 heartbeater.cc:347] Connected to a master server at 127.15.250.190:38697
I20260808 07:03:08.319542 22475 heartbeater.cc:464] Registering TS with master...
I20260808 07:03:08.319795 22475 heartbeater.cc:511] Master 127.15.250.190:38697 requested a full tablet report, sending...
I20260808 07:03:08.320189 22139 ts_manager.cc:194] Registered new tserver with Master: 84283ed4c56d4f92a197f35dfd077790 (127.15.250.130:32871)
I20260808 07:03:08.320645 22139 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.130:55245
W20260808 07:03:08.401724 22478 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 07:03:08.401978 22478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:08.402009 22478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:08.402050 22478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:08.403326 22478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:08.403404 22478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.131
I20260808 07:03:08.404807 22478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:39149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.250.131
--webserver_port=0
--tserver_master_addrs=127.15.250.190:38697
--never_fsync=true
--heap_profile_path=/tmp/kudu.22478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.131
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:08.405078 22478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:08.405406 22478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:08.411327 22483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:08.411315 22486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:08.411319 22484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:08.411700 22478 server_base.cc:1034] running on GCE node
I20260808 07:03:08.411942 22478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:08.412209 22478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:08.413419 22478 hybrid_clock.cc:648] HybridClock initialized: now 1786172588413401 us; error 32 us; skew 500 ppm
I20260808 07:03:08.414705 22478 webserver.cc:460] Webserver started at http://127.15.250.131:34033/ using document root <none> and password file <none>
I20260808 07:03:08.415002 22478 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:08.415078 22478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:08.415251 22478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:08.416323 22478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "72a1af17c1e6408c9ad06cdeea0923ff"
format_stamp: "Formatted at 2026-08-08 07:03:08 on dist-test-slave-0b58"
I20260808 07:03:08.416709 22478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "72a1af17c1e6408c9ad06cdeea0923ff"
format_stamp: "Formatted at 2026-08-08 07:03:08 on dist-test-slave-0b58"
I20260808 07:03:08.418231 22478 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 07:03:08.419006 22492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:08.419235 22478 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 07:03:08.419339 22478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "72a1af17c1e6408c9ad06cdeea0923ff"
format_stamp: "Formatted at 2026-08-08 07:03:08 on dist-test-slave-0b58"
I20260808 07:03:08.419427 22478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:08.438341 22478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:08.438697 22478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:08.438863 22478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:08.439121 22478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:03:08.439494 22478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:03:08.439553 22478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:08.439595 22478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:03:08.439643 22478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:08.445549 22478 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.131:43359
I20260808 07:03:08.445604 22610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.131:43359 every 8 connection(s)
I20260808 07:03:08.445971 22478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260808 07:03:08.452050 22612 heartbeater.cc:347] Connected to a master server at 127.15.250.190:38697
I20260808 07:03:08.452143 22612 heartbeater.cc:464] Registering TS with master...
I20260808 07:03:08.452349 22612 heartbeater.cc:511] Master 127.15.250.190:38697 requested a full tablet report, sending...
I20260808 07:03:08.452761 22139 ts_manager.cc:194] Registered new tserver with Master: 72a1af17c1e6408c9ad06cdeea0923ff (127.15.250.131:43359)
I20260808 07:03:08.453094 22139 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.131:45037
I20260808 07:03:08.453689 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 22478
I20260808 07:03:08.453811 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260808 07:03:08.454963 16362 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 07:03:08.467711 22139 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49048:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20260808 07:03:08.469460 22139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 07:03:08.477185 22404 tablet_service.cc:1467] Processing CreateTablet for tablet d52dcdb9ea744dc9b09723dfe25b8e7a (DEFAULT_TABLE table=TestTable [id=f4f71a5e6f6a4d1fb29e409416ff1e33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:03:08.477188 22266 tablet_service.cc:1467] Processing CreateTablet for tablet d52dcdb9ea744dc9b09723dfe25b8e7a (DEFAULT_TABLE table=TestTable [id=f4f71a5e6f6a4d1fb29e409416ff1e33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:03:08.477207 22543 tablet_service.cc:1467] Processing CreateTablet for tablet d52dcdb9ea744dc9b09723dfe25b8e7a (DEFAULT_TABLE table=TestTable [id=f4f71a5e6f6a4d1fb29e409416ff1e33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:03:08.477614 22266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d52dcdb9ea744dc9b09723dfe25b8e7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:08.477631 22543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d52dcdb9ea744dc9b09723dfe25b8e7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:08.477626 22404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d52dcdb9ea744dc9b09723dfe25b8e7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:08.480430 22626 tablet_bootstrap.cc:492] T d52dcdb9ea744dc9b09723dfe25b8e7a P 32dd316e9a274f2183a23289815cb166: Bootstrap starting.
I20260808 07:03:08.480997 22628 tablet_bootstrap.cc:492] T d52dcdb9ea744dc9b09723dfe25b8e7a P 84283ed4c56d4f92a197f35dfd077790: Bootstrap starting.
I20260808 07:03:08.481175 22626 tablet_bootstrap.cc:654] T d52dcdb9ea744dc9b09723dfe25b8e7a P 32dd316e9a274f2183a23289815cb166: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:08.481186 22627 tablet_bootstrap.cc:492] T d52dcdb9ea744dc9b09723dfe25b8e7a P 72a1af17c1e6408c9ad06cdeea0923ff: Bootstrap starting.
I20260808 07:03:08.481529 22626 log.cc:824] T d52dcdb9ea744dc9b09723dfe25b8e7a P 32dd316e9a274f2183a23289815cb166: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:08.481861 22627 tablet_bootstrap.cc:654] T d52dcdb9ea744dc9b09723dfe25b8e7a P 72a1af17c1e6408c9ad06cdeea0923ff: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:08.482172 22627 log.cc:824] T d52dcdb9ea744dc9b09723dfe25b8e7a P 72a1af17c1e6408c9ad06cdeea0923ff: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:08.482259 22626 tablet_bootstrap.cc:492] T d52dcdb9ea744dc9b09723dfe25b8e7a P 32dd316e9a274f2183a23289815cb166: No bootstrap required, opened a new log
I20260808 07:03:08.482316 22626 ts_tablet_manager.cc:1397] T d52dcdb9ea744dc9b09723dfe25b8e7a P 32dd316e9a274f2183a23289815cb166: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:03:08.482903 22627 tablet_bootstrap.cc:492] T d52dcdb9ea744dc9b09723dfe25b8e7a P 72a1af17c1e6408c9ad06cdeea0923ff: No bootstrap required, opened a new log
I20260808 07:03:08.482919 22628 tablet_bootstrap.cc:654] T d52dcdb9ea744dc9b09723dfe25b8e7a P 84283ed4c56d4f92a197f35dfd077790: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:08.482977 22627 ts_tablet_manager.cc:1397] T d52dcdb9ea744dc9b09723dfe25b8e7a P 72a1af17c1e6408c9ad06cdeea0923ff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:03:08.483268 22628 log.cc:824] T d52dcdb9ea744dc9b09723dfe25b8e7a P 84283ed4c56d4f92a197f35dfd077790: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:08.484001 22628 tablet_bootstrap.cc:492] T d52dcdb9ea744dc9b09723dfe25b8e7a P 84283ed4c56d4f92a197f35dfd077790: No bootstrap required, opened a new log
I20260808 07:03:08.484084 22628 ts_tablet_manager.cc:1397] T d52dcdb9ea744dc9b09723dfe25b8e7a P 84283ed4c56d4f92a197f35dfd077790: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260808 07:03:08.484982 22626 raft_consensus.cc:348] T d52dcdb9ea744dc9b09723dfe25b8e7a P 32dd316e9a274f2183a23289815cb166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a1af17c1e6408c9ad06cdeea0923ff" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 43359 } } peers { permanent_uuid: "84283ed4c56d4f92a197f35dfd077790" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 32871 } } peers { permanent_uuid: "32dd316e9a274f2183a23289815cb166" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 44379 } }
I20260808 07:03:08.485160 22626 raft_consensus.cc:729] T d52dcdb9ea744dc9b09723dfe25b8e7a P 32dd316e9a274f2183a23289815cb166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32dd316e9a274f2183a23289815cb166, State: Initialized, Role: FOLLOWER
I20260808 07:03:08.485244 22626 consensus_queue.cc:260] T d52dcdb9ea744dc9b09723dfe25b8e7a P 32dd316e9a274f2183a23289815cb166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a1af17c1e6408c9ad06cdeea0923ff" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 43359 } } peers { permanent_uuid: "84283ed4c56d4f92a197f35dfd077790" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 32871 } } peers { permanent_uuid: "32dd316e9a274f2183a23289815cb166" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 44379 } }
I20260808 07:03:08.485518 22626 ts_tablet_manager.cc:1428] T d52dcdb9ea744dc9b09723dfe25b8e7a P 32dd316e9a274f2183a23289815cb166: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 07:03:08.485545 22335 heartbeater.cc:503] Master 127.15.250.190:38697 was elected leader, sending a full tablet report...
I20260808 07:03:08.485661 22627 raft_consensus.cc:348] T d52dcdb9ea744dc9b09723dfe25b8e7a P 72a1af17c1e6408c9ad06cdeea0923ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a1af17c1e6408c9ad06cdeea0923ff" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 43359 } } peers { permanent_uuid: "84283ed4c56d4f92a197f35dfd077790" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 32871 } } peers { permanent_uuid: "32dd316e9a274f2183a23289815cb166" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 44379 } }
I20260808 07:03:08.485833 22627 raft_consensus.cc:729] T d52dcdb9ea744dc9b09723dfe25b8e7a P 72a1af17c1e6408c9ad06cdeea0923ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72a1af17c1e6408c9ad06cdeea0923ff, State: Initialized, Role: FOLLOWER
I20260808 07:03:08.485927 22627 consensus_queue.cc:260] T d52dcdb9ea744dc9b09723dfe25b8e7a P 72a1af17c1e6408c9ad06cdeea0923ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a1af17c1e6408c9ad06cdeea0923ff" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 43359 } } peers { permanent_uuid: "84283ed4c56d4f92a197f35dfd077790" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 32871 } } peers { permanent_uuid: "32dd316e9a274f2183a23289815cb166" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 44379 } }
I20260808 07:03:08.486207 22627 ts_tablet_manager.cc:1428] T d52dcdb9ea744dc9b09723dfe25b8e7a P 72a1af17c1e6408c9ad06cdeea0923ff: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260808 07:03:08.486232 22612 heartbeater.cc:503] Master 127.15.250.190:38697 was elected leader, sending a full tablet report...
I20260808 07:03:08.486831 22628 raft_consensus.cc:348] T d52dcdb9ea744dc9b09723dfe25b8e7a P 84283ed4c56d4f92a197f35dfd077790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a1af17c1e6408c9ad06cdeea0923ff" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 43359 } } peers { permanent_uuid: "84283ed4c56d4f92a197f35dfd077790" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 32871 } } peers { permanent_uuid: "32dd316e9a274f2183a23289815cb166" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 44379 } }
I20260808 07:03:08.487012 22628 raft_consensus.cc:729] T d52dcdb9ea744dc9b09723dfe25b8e7a P 84283ed4c56d4f92a197f35dfd077790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84283ed4c56d4f92a197f35dfd077790, State: Initialized, Role: FOLLOWER
I20260808 07:03:08.487260 22628 consensus_queue.cc:260] T d52dcdb9ea744dc9b09723dfe25b8e7a P 84283ed4c56d4f92a197f35dfd077790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a1af17c1e6408c9ad06cdeea0923ff" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 43359 } } peers { permanent_uuid: "84283ed4c56d4f92a197f35dfd077790" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 32871 } } peers { permanent_uuid: "32dd316e9a274f2183a23289815cb166" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 44379 } }
I20260808 07:03:08.487563 22628 ts_tablet_manager.cc:1428] T d52dcdb9ea744dc9b09723dfe25b8e7a P 84283ed4c56d4f92a197f35dfd077790: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 07:03:08.487682 22475 heartbeater.cc:503] Master 127.15.250.190:38697 was elected leader, sending a full tablet report...
I20260808 07:03:08.489166 16362 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 07:03:08.489746 16362 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 32dd316e9a274f2183a23289815cb166 to finish bootstrapping
I20260808 07:03:08.493899 16362 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 84283ed4c56d4f92a197f35dfd077790 to finish bootstrapping
I20260808 07:03:08.497553 16362 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 72a1af17c1e6408c9ad06cdeea0923ff to finish bootstrapping
I20260808 07:03:08.504833 22425 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d52dcdb9ea744dc9b09723dfe25b8e7a"
dest_uuid: "84283ed4c56d4f92a197f35dfd077790"
from {username='slave'} at 127.0.0.1:51142
I20260808 07:03:08.504973 22425 raft_consensus.cc:482] T d52dcdb9ea744dc9b09723dfe25b8e7a P 84283ed4c56d4f92a197f35dfd077790 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 07:03:08.505041 22425 raft_consensus.cc:3037] T d52dcdb9ea744dc9b09723dfe25b8e7a P 84283ed4c56d4f92a197f35dfd077790 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:08.505797 22425 raft_consensus.cc:504] T d52dcdb9ea744dc9b09723dfe25b8e7a P 84283ed4c56d4f92a197f35dfd077790 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a1af17c1e6408c9ad06cdeea0923ff" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 43359 } } peers { permanent_uuid: "84283ed4c56d4f92a197f35dfd077790" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 32871 } } peers { permanent_uuid: "32dd316e9a274f2183a23289815cb166" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 44379 } }
I20260808 07:03:08.506114 22425 leader_election.cc:288] T d52dcdb9ea744dc9b09723dfe25b8e7a P 84283ed4c56d4f92a197f35dfd077790 [CANDIDATE]: Term 1 election: Requested vote from peers 72a1af17c1e6408c9ad06cdeea0923ff (127.15.250.131:43359), 32dd316e9a274f2183a23289815cb166 (127.15.250.129:44379)
I20260808 07:03:08.510058 22287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d52dcdb9ea744dc9b09723dfe25b8e7a" candidate_uuid: "84283ed4c56d4f92a197f35dfd077790" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "32dd316e9a274f2183a23289815cb166"
I20260808 07:03:08.510056 22565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d52dcdb9ea744dc9b09723dfe25b8e7a" candidate_uuid: "84283ed4c56d4f92a197f35dfd077790" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "72a1af17c1e6408c9ad06cdeea0923ff"
I20260808 07:03:08.510202 22565 raft_consensus.cc:3037] T d52dcdb9ea744dc9b09723dfe25b8e7a P 72a1af17c1e6408c9ad06cdeea0923ff [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:08.510202 22287 raft_consensus.cc:3037] T d52dcdb9ea744dc9b09723dfe25b8e7a P 32dd316e9a274f2183a23289815cb166 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:08.511113 22287 raft_consensus.cc:2454] T d52dcdb9ea744dc9b09723dfe25b8e7a P 32dd316e9a274f2183a23289815cb166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84283ed4c56d4f92a197f35dfd077790 in term 1.
I20260808 07:03:08.511103 22565 raft_consensus.cc:2454] T d52dcdb9ea744dc9b09723dfe25b8e7a P 72a1af17c1e6408c9ad06cdeea0923ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84283ed4c56d4f92a197f35dfd077790 in term 1.
I20260808 07:03:08.511405 22358 leader_election.cc:302] T d52dcdb9ea744dc9b09723dfe25b8e7a P 84283ed4c56d4f92a197f35dfd077790 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72a1af17c1e6408c9ad06cdeea0923ff, 84283ed4c56d4f92a197f35dfd077790; no voters:
I20260808 07:03:08.511600 22634 raft_consensus.cc:2781] T d52dcdb9ea744dc9b09723dfe25b8e7a P 84283ed4c56d4f92a197f35dfd077790 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:08.511834 22634 raft_consensus.cc:686] T d52dcdb9ea744dc9b09723dfe25b8e7a P 84283ed4c56d4f92a197f35dfd077790 [term 1 LEADER]: Becoming Leader. State: Replica: 84283ed4c56d4f92a197f35dfd077790, State: Running, Role: LEADER
I20260808 07:03:08.511969 22634 consensus_queue.cc:237] T d52dcdb9ea744dc9b09723dfe25b8e7a P 84283ed4c56d4f92a197f35dfd077790 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a1af17c1e6408c9ad06cdeea0923ff" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 43359 } } peers { permanent_uuid: "84283ed4c56d4f92a197f35dfd077790" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 32871 } } peers { permanent_uuid: "32dd316e9a274f2183a23289815cb166" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 44379 } }
I20260808 07:03:08.512952 22138 catalog_manager.cc:5495] T d52dcdb9ea744dc9b09723dfe25b8e7a P 84283ed4c56d4f92a197f35dfd077790 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84283ed4c56d4f92a197f35dfd077790 (127.15.250.130). New cstate: current_term: 1 leader_uuid: "84283ed4c56d4f92a197f35dfd077790" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a1af17c1e6408c9ad06cdeea0923ff" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 43359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84283ed4c56d4f92a197f35dfd077790" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 32871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32dd316e9a274f2183a23289815cb166" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 44379 } health_report { overall_health: UNKNOWN } } }
I20260808 07:03:08.517446 22287 raft_consensus.cc:1261] T d52dcdb9ea744dc9b09723dfe25b8e7a P 32dd316e9a274f2183a23289815cb166 [term 1 FOLLOWER]: Refusing update from remote peer 84283ed4c56d4f92a197f35dfd077790: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 07:03:08.517535 22565 raft_consensus.cc:1261] T d52dcdb9ea744dc9b09723dfe25b8e7a P 72a1af17c1e6408c9ad06cdeea0923ff [term 1 FOLLOWER]: Refusing update from remote peer 84283ed4c56d4f92a197f35dfd077790: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 07:03:08.517849 22637 consensus_queue.cc:1035] T d52dcdb9ea744dc9b09723dfe25b8e7a P 84283ed4c56d4f92a197f35dfd077790 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32dd316e9a274f2183a23289815cb166" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 44379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:03:08.517978 22634 consensus_queue.cc:1035] T d52dcdb9ea744dc9b09723dfe25b8e7a P 84283ed4c56d4f92a197f35dfd077790 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72a1af17c1e6408c9ad06cdeea0923ff" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 43359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:03:08.520403 22639 mvcc.cc:205] Tried to move back new op lower bound from 7316162922564403200 to 7316162922545901568. Current Snapshot: MvccSnapshot[applied={T|T < 7316162922564403200}]
I20260808 07:03:08.521344 22641 mvcc.cc:205] Tried to move back new op lower bound from 7316162922564403200 to 7316162922545901568. Current Snapshot: MvccSnapshot[applied={T|T < 7316162922564403200}]
I20260808 07:03:08.529109 22425 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d52dcdb9ea744dc9b09723dfe25b8e7a"
dest_uuid: "84283ed4c56d4f92a197f35dfd077790"
from {username='slave'} at 127.0.0.1:51142
I20260808 07:03:08.529220 22425 raft_consensus.cc:580] T d52dcdb9ea744dc9b09723dfe25b8e7a P 84283ed4c56d4f92a197f35dfd077790 [term 1 LEADER]: Received request to step down
I20260808 07:03:08.529251 22425 raft_consensus.cc:3032] T d52dcdb9ea744dc9b09723dfe25b8e7a P 84283ed4c56d4f92a197f35dfd077790 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 07:03:08.529280 22425 raft_consensus.cc:729] T d52dcdb9ea744dc9b09723dfe25b8e7a P 84283ed4c56d4f92a197f35dfd077790 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 84283ed4c56d4f92a197f35dfd077790, State: Running, Role: LEADER
I20260808 07:03:08.529325 22425 consensus_queue.cc:260] T d52dcdb9ea744dc9b09723dfe25b8e7a P 84283ed4c56d4f92a197f35dfd077790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a1af17c1e6408c9ad06cdeea0923ff" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 43359 } } peers { permanent_uuid: "84283ed4c56d4f92a197f35dfd077790" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 32871 } } peers { permanent_uuid: "32dd316e9a274f2183a23289815cb166" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 44379 } }
I20260808 07:03:08.529470 22425 raft_consensus.cc:3037] T d52dcdb9ea744dc9b09723dfe25b8e7a P 84283ed4c56d4f92a197f35dfd077790 [term 1 FOLLOWER]: Advancing to term 2
I20260808 07:03:08.530140 22425 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d52dcdb9ea744dc9b09723dfe25b8e7a"
dest_uuid: "84283ed4c56d4f92a197f35dfd077790"
from {username='slave'} at 127.0.0.1:51142
I20260808 07:03:08.530215 22425 raft_consensus.cc:573] T d52dcdb9ea744dc9b09723dfe25b8e7a P 84283ed4c56d4f92a197f35dfd077790 [term 2 FOLLOWER]: Rejecting request to step down while not leader
W20260808 07:03:08.530804 22384 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51142: Illegal state: replica 84283ed4c56d4f92a197f35dfd077790 is not leader of this config: current role FOLLOWER
I20260808 07:03:08.531215 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 22202
I20260808 07:03:08.540185 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 22340
I20260808 07:03:08.546108 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 22478
I20260808 07:03:08.551369 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 22106
2026-08-08T07:03:08Z chronyd exiting
[ OK ] RaftConsensusElectionITest.LeaderStepDown (715 ms)
[ RUN ] RaftConsensusElectionITest.StepDownWithSlowFollower
I20260808 07:03:08.572156 16362 test_util.cc:268] Using random seed: -1230619145
I20260808 07:03:08.572762 16362 ts_itest-base.cc:112] Starting cluster with:
I20260808 07:03:08.572821 16362 ts_itest-base.cc:113] --------------
I20260808 07:03:08.572840 16362 ts_itest-base.cc:114] 3 tablet servers
I20260808 07:03:08.572860 16362 ts_itest-base.cc:115] 3 replicas per TS
I20260808 07:03:08.572885 16362 ts_itest-base.cc:116] --------------
2026-08-08T07:03:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:03:08Z Disabled control of system clock
I20260808 07:03:08.581655 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.250.190:43337
--webserver_interface=127.15.250.190
--webserver_port=0
--builtin_ntp_servers=127.15.250.148:45831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.250.190:43337
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20260808 07:03:08.663084 22654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:08.663352 22654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:08.663389 22654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:08.664601 22654 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:03:08.664680 22654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:08.664705 22654 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:03:08.664724 22654 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:03:08.666167 22654 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:45831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.15.250.190:43337
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.250.190:43337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.250.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.22654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:08.666468 22654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:08.666785 22654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 07:03:08.672088 22654 server_base.cc:1034] running on GCE node
W20260808 07:03:08.672079 22660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:08.672111 22663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:08.672073 22661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:08.672631 22654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:08.673051 22654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:08.674216 22654 hybrid_clock.cc:648] HybridClock initialized: now 1786172588674191 us; error 42 us; skew 500 ppm
I20260808 07:03:08.675546 22654 webserver.cc:460] Webserver started at http://127.15.250.190:46453/ using document root <none> and password file <none>
I20260808 07:03:08.675820 22654 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:08.675908 22654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:08.676066 22654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:08.677152 22654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "caaaca0f7f2847cb8987074d188868b4"
format_stamp: "Formatted at 2026-08-08 07:03:08 on dist-test-slave-0b58"
I20260808 07:03:08.677534 22654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "caaaca0f7f2847cb8987074d188868b4"
format_stamp: "Formatted at 2026-08-08 07:03:08 on dist-test-slave-0b58"
I20260808 07:03:08.679033 22654 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 07:03:08.679941 22669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:08.680168 22654 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 07:03:08.680320 22654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
uuid: "caaaca0f7f2847cb8987074d188868b4"
format_stamp: "Formatted at 2026-08-08 07:03:08 on dist-test-slave-0b58"
I20260808 07:03:08.680414 22654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:08.690891 22654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:08.691320 22654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:08.691488 22654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:08.695727 22654 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.190:43337
I20260808 07:03:08.695806 22724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.190:43337 every 8 connection(s)
I20260808 07:03:08.696165 22654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260808 07:03:08.696484 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 22654
I20260808 07:03:08.696583 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260808 07:03:08.697443 22725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:08.700436 22725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caaaca0f7f2847cb8987074d188868b4: Bootstrap starting.
I20260808 07:03:08.701115 22725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P caaaca0f7f2847cb8987074d188868b4: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:08.701485 22725 log.cc:824] T 00000000000000000000000000000000 P caaaca0f7f2847cb8987074d188868b4: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:08.702212 22725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caaaca0f7f2847cb8987074d188868b4: No bootstrap required, opened a new log
I20260808 07:03:08.703712 22725 raft_consensus.cc:348] T 00000000000000000000000000000000 P caaaca0f7f2847cb8987074d188868b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caaaca0f7f2847cb8987074d188868b4" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 43337 } }
I20260808 07:03:08.703886 22725 raft_consensus.cc:374] T 00000000000000000000000000000000 P caaaca0f7f2847cb8987074d188868b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:03:08.703936 22725 raft_consensus.cc:729] T 00000000000000000000000000000000 P caaaca0f7f2847cb8987074d188868b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caaaca0f7f2847cb8987074d188868b4, State: Initialized, Role: FOLLOWER
I20260808 07:03:08.704071 22725 consensus_queue.cc:260] T 00000000000000000000000000000000 P caaaca0f7f2847cb8987074d188868b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caaaca0f7f2847cb8987074d188868b4" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 43337 } }
I20260808 07:03:08.704152 22725 raft_consensus.cc:388] T 00000000000000000000000000000000 P caaaca0f7f2847cb8987074d188868b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:03:08.704192 22725 raft_consensus.cc:482] T 00000000000000000000000000000000 P caaaca0f7f2847cb8987074d188868b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:03:08.704241 22725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P caaaca0f7f2847cb8987074d188868b4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:08.705191 22725 raft_consensus.cc:504] T 00000000000000000000000000000000 P caaaca0f7f2847cb8987074d188868b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caaaca0f7f2847cb8987074d188868b4" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 43337 } }
I20260808 07:03:08.705338 22725 leader_election.cc:302] T 00000000000000000000000000000000 P caaaca0f7f2847cb8987074d188868b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: caaaca0f7f2847cb8987074d188868b4; no voters:
I20260808 07:03:08.705529 22725 leader_election.cc:288] T 00000000000000000000000000000000 P caaaca0f7f2847cb8987074d188868b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:03:08.705631 22730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P caaaca0f7f2847cb8987074d188868b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:08.705823 22730 raft_consensus.cc:686] T 00000000000000000000000000000000 P caaaca0f7f2847cb8987074d188868b4 [term 1 LEADER]: Becoming Leader. State: Replica: caaaca0f7f2847cb8987074d188868b4, State: Running, Role: LEADER
I20260808 07:03:08.705839 22725 sys_catalog.cc:564] T 00000000000000000000000000000000 P caaaca0f7f2847cb8987074d188868b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:03:08.705925 22730 consensus_queue.cc:237] T 00000000000000000000000000000000 P caaaca0f7f2847cb8987074d188868b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caaaca0f7f2847cb8987074d188868b4" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 43337 } }
I20260808 07:03:08.706507 22731 sys_catalog.cc:455] T 00000000000000000000000000000000 P caaaca0f7f2847cb8987074d188868b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "caaaca0f7f2847cb8987074d188868b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caaaca0f7f2847cb8987074d188868b4" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 43337 } } }
I20260808 07:03:08.706514 22732 sys_catalog.cc:455] T 00000000000000000000000000000000 P caaaca0f7f2847cb8987074d188868b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader caaaca0f7f2847cb8987074d188868b4. Latest consensus state: current_term: 1 leader_uuid: "caaaca0f7f2847cb8987074d188868b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caaaca0f7f2847cb8987074d188868b4" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 43337 } } }
I20260808 07:03:08.706615 22731 sys_catalog.cc:458] T 00000000000000000000000000000000 P caaaca0f7f2847cb8987074d188868b4 [sys.catalog]: This master's current role is: LEADER
I20260808 07:03:08.706629 22732 sys_catalog.cc:458] T 00000000000000000000000000000000 P caaaca0f7f2847cb8987074d188868b4 [sys.catalog]: This master's current role is: LEADER
I20260808 07:03:08.706837 22739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:03:08.707363 22739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:03:08.708920 22739 catalog_manager.cc:1347] Generated new cluster ID: 9f0cf093fb254af2aadce8a749aa307e
I20260808 07:03:08.708977 22739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:03:08.731570 22739 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:03:08.732372 22739 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:03:08.738976 22739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P caaaca0f7f2847cb8987074d188868b4: Generated new TSK 0
I20260808 07:03:08.739180 22739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:03:08.747263 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.129:0
--local_ip_for_outbound_sockets=127.15.250.129
--webserver_interface=127.15.250.129
--webserver_port=0
--tserver_master_addrs=127.15.250.190:43337
--builtin_ntp_servers=127.15.250.148:45831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--consensus_rpc_timeout_ms=15000 with env {}
W20260808 07:03:08.831197 22749 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 07:03:08.831467 22749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:08.831498 22749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:08.831543 22749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:08.832855 22749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:08.832939 22749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.129
I20260808 07:03:08.834329 22749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:45831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=15000
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.250.129
--webserver_port=0
--tserver_master_addrs=127.15.250.190:43337
--never_fsync=true
--heap_profile_path=/tmp/kudu.22749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.129
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:08.834602 22749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:08.834923 22749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:08.840035 22755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:08.840096 22758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:08.840045 22756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:08.840507 22749 server_base.cc:1034] running on GCE node
I20260808 07:03:08.840694 22749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:08.840894 22749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:08.842028 22749 hybrid_clock.cc:648] HybridClock initialized: now 1786172588842014 us; error 29 us; skew 500 ppm
I20260808 07:03:08.843629 22749 webserver.cc:460] Webserver started at http://127.15.250.129:45273/ using document root <none> and password file <none>
I20260808 07:03:08.843842 22749 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:08.843892 22749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:08.843986 22749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:08.845049 22749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "4c604fa7ff214d748eb788f7b7a9f695"
format_stamp: "Formatted at 2026-08-08 07:03:08 on dist-test-slave-0b58"
I20260808 07:03:08.845373 22749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "4c604fa7ff214d748eb788f7b7a9f695"
format_stamp: "Formatted at 2026-08-08 07:03:08 on dist-test-slave-0b58"
I20260808 07:03:08.846642 22749 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 07:03:08.847345 22764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:08.847503 22749 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:03:08.847590 22749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4c604fa7ff214d748eb788f7b7a9f695"
format_stamp: "Formatted at 2026-08-08 07:03:08 on dist-test-slave-0b58"
I20260808 07:03:08.847645 22749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:08.863463 22749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:08.863752 22749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:08.863862 22749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:08.864048 22749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:03:08.864501 22749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:03:08.864552 22749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:08.864575 22749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:03:08.864621 22749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:08.870478 22749 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.129:40993
I20260808 07:03:08.870575 22882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.129:40993 every 8 connection(s)
I20260808 07:03:08.870923 22749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260808 07:03:08.872982 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 22749
I20260808 07:03:08.873109 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260808 07:03:08.874369 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.130:0
--local_ip_for_outbound_sockets=127.15.250.130
--webserver_interface=127.15.250.130
--webserver_port=0
--tserver_master_addrs=127.15.250.190:43337
--builtin_ntp_servers=127.15.250.148:45831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--consensus_rpc_timeout_ms=15000 with env {}
I20260808 07:03:08.877542 22883 heartbeater.cc:347] Connected to a master server at 127.15.250.190:43337
I20260808 07:03:08.877661 22883 heartbeater.cc:464] Registering TS with master...
I20260808 07:03:08.877861 22883 heartbeater.cc:511] Master 127.15.250.190:43337 requested a full tablet report, sending...
I20260808 07:03:08.878494 22688 ts_manager.cc:194] Registered new tserver with Master: 4c604fa7ff214d748eb788f7b7a9f695 (127.15.250.129:40993)
I20260808 07:03:08.879206 22688 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.129:34253
W20260808 07:03:08.958489 22886 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 07:03:08.958722 22886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:08.958740 22886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:08.958760 22886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:08.959957 22886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:08.960011 22886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.130
I20260808 07:03:08.961375 22886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:45831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=15000
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.250.130
--webserver_port=0
--tserver_master_addrs=127.15.250.190:43337
--never_fsync=true
--heap_profile_path=/tmp/kudu.22886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.130
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:08.961580 22886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:08.961846 22886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 07:03:08.967228 22886 server_base.cc:1034] running on GCE node
W20260808 07:03:08.967295 22892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:08.967285 22891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:08.967285 22895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:08.967684 22886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:08.967957 22886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:08.969127 22886 hybrid_clock.cc:648] HybridClock initialized: now 1786172588969110 us; error 31 us; skew 500 ppm
I20260808 07:03:08.970374 22886 webserver.cc:460] Webserver started at http://127.15.250.130:44971/ using document root <none> and password file <none>
I20260808 07:03:08.970623 22886 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:08.970696 22886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:08.970866 22886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:08.972043 22886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "82f384a95529443683b2870ca150ce97"
format_stamp: "Formatted at 2026-08-08 07:03:08 on dist-test-slave-0b58"
I20260808 07:03:08.972453 22886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "82f384a95529443683b2870ca150ce97"
format_stamp: "Formatted at 2026-08-08 07:03:08 on dist-test-slave-0b58"
I20260808 07:03:08.973938 22886 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 07:03:08.974743 22901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:08.974944 22886 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 07:03:08.975049 22886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "82f384a95529443683b2870ca150ce97"
format_stamp: "Formatted at 2026-08-08 07:03:08 on dist-test-slave-0b58"
I20260808 07:03:08.975139 22886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:08.989756 22886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:08.990159 22886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:08.990327 22886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:08.990585 22886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:03:08.990942 22886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:03:08.991008 22886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:08.991055 22886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:03:08.991079 22886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:08.997116 22886 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.130:34447
I20260808 07:03:08.997203 23021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.130:34447 every 8 connection(s)
I20260808 07:03:08.997548 22886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260808 07:03:09.000856 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 22886
I20260808 07:03:09.000978 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260808 07:03:09.002172 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.131:0
--local_ip_for_outbound_sockets=127.15.250.131
--webserver_interface=127.15.250.131
--webserver_port=0
--tserver_master_addrs=127.15.250.190:43337
--builtin_ntp_servers=127.15.250.148:45831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--consensus_rpc_timeout_ms=15000 with env {}
I20260808 07:03:09.003925 23022 heartbeater.cc:347] Connected to a master server at 127.15.250.190:43337
I20260808 07:03:09.004051 23022 heartbeater.cc:464] Registering TS with master...
I20260808 07:03:09.004235 23022 heartbeater.cc:511] Master 127.15.250.190:43337 requested a full tablet report, sending...
I20260808 07:03:09.004673 22688 ts_manager.cc:194] Registered new tserver with Master: 82f384a95529443683b2870ca150ce97 (127.15.250.130:34447)
I20260808 07:03:09.005076 22688 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.130:43077
W20260808 07:03:09.086109 23025 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 07:03:09.086347 23025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:09.086364 23025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:09.086380 23025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:09.087544 23025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:09.087597 23025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.131
I20260808 07:03:09.089025 23025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:45831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=15000
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.250.131
--webserver_port=0
--tserver_master_addrs=127.15.250.190:43337
--never_fsync=true
--heap_profile_path=/tmp/kudu.23025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.131
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:09.089221 23025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:09.089468 23025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:09.094661 23030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:09.094655 23033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:09.094656 23031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:09.094923 23025 server_base.cc:1034] running on GCE node
I20260808 07:03:09.095098 23025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:09.095292 23025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:09.096463 23025 hybrid_clock.cc:648] HybridClock initialized: now 1786172589096457 us; error 35 us; skew 500 ppm
I20260808 07:03:09.097792 23025 webserver.cc:460] Webserver started at http://127.15.250.131:41169/ using document root <none> and password file <none>
I20260808 07:03:09.098053 23025 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:09.098100 23025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:09.098192 23025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:09.099140 23025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "90a2711b680e473d9a1f95824ff5a118"
format_stamp: "Formatted at 2026-08-08 07:03:09 on dist-test-slave-0b58"
I20260808 07:03:09.099452 23025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "90a2711b680e473d9a1f95824ff5a118"
format_stamp: "Formatted at 2026-08-08 07:03:09 on dist-test-slave-0b58"
I20260808 07:03:09.100841 23025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 07:03:09.101562 23039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:09.101765 23025 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:03:09.101845 23025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "90a2711b680e473d9a1f95824ff5a118"
format_stamp: "Formatted at 2026-08-08 07:03:09 on dist-test-slave-0b58"
I20260808 07:03:09.101899 23025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:09.127818 23025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:09.128113 23025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:09.128255 23025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:09.128546 23025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:03:09.128891 23025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:03:09.128924 23025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:09.128978 23025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:03:09.129009 23025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:09.135288 23025 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.131:39363
I20260808 07:03:09.135378 23158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.131:39363 every 8 connection(s)
I20260808 07:03:09.135730 23025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260808 07:03:09.138605 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 23025
I20260808 07:03:09.138711 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260808 07:03:09.141860 23159 heartbeater.cc:347] Connected to a master server at 127.15.250.190:43337
I20260808 07:03:09.141952 23159 heartbeater.cc:464] Registering TS with master...
I20260808 07:03:09.142154 23159 heartbeater.cc:511] Master 127.15.250.190:43337 requested a full tablet report, sending...
I20260808 07:03:09.142552 22688 ts_manager.cc:194] Registered new tserver with Master: 90a2711b680e473d9a1f95824ff5a118 (127.15.250.131:39363)
I20260808 07:03:09.142927 22688 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.131:38305
I20260808 07:03:09.150851 16362 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 07:03:09.157104 22688 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48700:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20260808 07:03:09.158704 22688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 07:03:09.166044 22951 tablet_service.cc:1467] Processing CreateTablet for tablet ddcb98a444444aea9ea6e92101a89bbd (DEFAULT_TABLE table=TestTable [id=0fa92b38ac3d49b8a060d720e75489d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:03:09.166067 22815 tablet_service.cc:1467] Processing CreateTablet for tablet ddcb98a444444aea9ea6e92101a89bbd (DEFAULT_TABLE table=TestTable [id=0fa92b38ac3d49b8a060d720e75489d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:03:09.166416 22951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddcb98a444444aea9ea6e92101a89bbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:09.166330 23088 tablet_service.cc:1467] Processing CreateTablet for tablet ddcb98a444444aea9ea6e92101a89bbd (DEFAULT_TABLE table=TestTable [id=0fa92b38ac3d49b8a060d720e75489d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:03:09.166464 22815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddcb98a444444aea9ea6e92101a89bbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:09.166637 23088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddcb98a444444aea9ea6e92101a89bbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:09.168870 23173 tablet_bootstrap.cc:492] T ddcb98a444444aea9ea6e92101a89bbd P 82f384a95529443683b2870ca150ce97: Bootstrap starting.
I20260808 07:03:09.169642 23175 tablet_bootstrap.cc:492] T ddcb98a444444aea9ea6e92101a89bbd P 90a2711b680e473d9a1f95824ff5a118: Bootstrap starting.
I20260808 07:03:09.169689 23173 tablet_bootstrap.cc:654] T ddcb98a444444aea9ea6e92101a89bbd P 82f384a95529443683b2870ca150ce97: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:09.170054 23173 log.cc:824] T ddcb98a444444aea9ea6e92101a89bbd P 82f384a95529443683b2870ca150ce97: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:09.170318 23175 tablet_bootstrap.cc:654] T ddcb98a444444aea9ea6e92101a89bbd P 90a2711b680e473d9a1f95824ff5a118: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:09.170442 23174 tablet_bootstrap.cc:492] T ddcb98a444444aea9ea6e92101a89bbd P 4c604fa7ff214d748eb788f7b7a9f695: Bootstrap starting.
I20260808 07:03:09.170648 23175 log.cc:824] T ddcb98a444444aea9ea6e92101a89bbd P 90a2711b680e473d9a1f95824ff5a118: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:09.171159 23174 tablet_bootstrap.cc:654] T ddcb98a444444aea9ea6e92101a89bbd P 4c604fa7ff214d748eb788f7b7a9f695: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:09.171402 23175 tablet_bootstrap.cc:492] T ddcb98a444444aea9ea6e92101a89bbd P 90a2711b680e473d9a1f95824ff5a118: No bootstrap required, opened a new log
I20260808 07:03:09.171480 23175 ts_tablet_manager.cc:1397] T ddcb98a444444aea9ea6e92101a89bbd P 90a2711b680e473d9a1f95824ff5a118: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:03:09.171489 23174 log.cc:824] T ddcb98a444444aea9ea6e92101a89bbd P 4c604fa7ff214d748eb788f7b7a9f695: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:09.172019 23173 tablet_bootstrap.cc:492] T ddcb98a444444aea9ea6e92101a89bbd P 82f384a95529443683b2870ca150ce97: No bootstrap required, opened a new log
I20260808 07:03:09.172135 23173 ts_tablet_manager.cc:1397] T ddcb98a444444aea9ea6e92101a89bbd P 82f384a95529443683b2870ca150ce97: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 07:03:09.172528 23174 tablet_bootstrap.cc:492] T ddcb98a444444aea9ea6e92101a89bbd P 4c604fa7ff214d748eb788f7b7a9f695: No bootstrap required, opened a new log
I20260808 07:03:09.172612 23174 ts_tablet_manager.cc:1397] T ddcb98a444444aea9ea6e92101a89bbd P 4c604fa7ff214d748eb788f7b7a9f695: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:03:09.174363 23175 raft_consensus.cc:348] T ddcb98a444444aea9ea6e92101a89bbd P 90a2711b680e473d9a1f95824ff5a118 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f384a95529443683b2870ca150ce97" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34447 } } peers { permanent_uuid: "90a2711b680e473d9a1f95824ff5a118" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39363 } } peers { permanent_uuid: "4c604fa7ff214d748eb788f7b7a9f695" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 40993 } }
I20260808 07:03:09.174554 23175 raft_consensus.cc:729] T ddcb98a444444aea9ea6e92101a89bbd P 90a2711b680e473d9a1f95824ff5a118 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a2711b680e473d9a1f95824ff5a118, State: Initialized, Role: FOLLOWER
I20260808 07:03:09.174630 23175 consensus_queue.cc:260] T ddcb98a444444aea9ea6e92101a89bbd P 90a2711b680e473d9a1f95824ff5a118 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f384a95529443683b2870ca150ce97" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34447 } } peers { permanent_uuid: "90a2711b680e473d9a1f95824ff5a118" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39363 } } peers { permanent_uuid: "4c604fa7ff214d748eb788f7b7a9f695" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 40993 } }
I20260808 07:03:09.174659 23174 raft_consensus.cc:348] T ddcb98a444444aea9ea6e92101a89bbd P 4c604fa7ff214d748eb788f7b7a9f695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f384a95529443683b2870ca150ce97" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34447 } } peers { permanent_uuid: "90a2711b680e473d9a1f95824ff5a118" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39363 } } peers { permanent_uuid: "4c604fa7ff214d748eb788f7b7a9f695" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 40993 } }
I20260808 07:03:09.174837 23174 raft_consensus.cc:729] T ddcb98a444444aea9ea6e92101a89bbd P 4c604fa7ff214d748eb788f7b7a9f695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c604fa7ff214d748eb788f7b7a9f695, State: Initialized, Role: FOLLOWER
I20260808 07:03:09.174758 23173 raft_consensus.cc:348] T ddcb98a444444aea9ea6e92101a89bbd P 82f384a95529443683b2870ca150ce97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f384a95529443683b2870ca150ce97" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34447 } } peers { permanent_uuid: "90a2711b680e473d9a1f95824ff5a118" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39363 } } peers { permanent_uuid: "4c604fa7ff214d748eb788f7b7a9f695" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 40993 } }
I20260808 07:03:09.174923 23175 ts_tablet_manager.cc:1428] T ddcb98a444444aea9ea6e92101a89bbd P 90a2711b680e473d9a1f95824ff5a118: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260808 07:03:09.174932 23173 raft_consensus.cc:729] T ddcb98a444444aea9ea6e92101a89bbd P 82f384a95529443683b2870ca150ce97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f384a95529443683b2870ca150ce97, State: Initialized, Role: FOLLOWER
I20260808 07:03:09.174934 23174 consensus_queue.cc:260] T ddcb98a444444aea9ea6e92101a89bbd P 4c604fa7ff214d748eb788f7b7a9f695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f384a95529443683b2870ca150ce97" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34447 } } peers { permanent_uuid: "90a2711b680e473d9a1f95824ff5a118" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39363 } } peers { permanent_uuid: "4c604fa7ff214d748eb788f7b7a9f695" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 40993 } }
I20260808 07:03:09.175047 23159 heartbeater.cc:503] Master 127.15.250.190:43337 was elected leader, sending a full tablet report...
I20260808 07:03:09.175235 23174 ts_tablet_manager.cc:1428] T ddcb98a444444aea9ea6e92101a89bbd P 4c604fa7ff214d748eb788f7b7a9f695: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 07:03:09.175294 22883 heartbeater.cc:503] Master 127.15.250.190:43337 was elected leader, sending a full tablet report...
I20260808 07:03:09.175004 23173 consensus_queue.cc:260] T ddcb98a444444aea9ea6e92101a89bbd P 82f384a95529443683b2870ca150ce97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f384a95529443683b2870ca150ce97" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34447 } } peers { permanent_uuid: "90a2711b680e473d9a1f95824ff5a118" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39363 } } peers { permanent_uuid: "4c604fa7ff214d748eb788f7b7a9f695" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 40993 } }
I20260808 07:03:09.175604 23022 heartbeater.cc:503] Master 127.15.250.190:43337 was elected leader, sending a full tablet report...
I20260808 07:03:09.175806 23173 ts_tablet_manager.cc:1428] T ddcb98a444444aea9ea6e92101a89bbd P 82f384a95529443683b2870ca150ce97: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260808 07:03:09.181792 16362 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 07:03:09.182360 16362 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4c604fa7ff214d748eb788f7b7a9f695 to finish bootstrapping
I20260808 07:03:09.186499 16362 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 82f384a95529443683b2870ca150ce97 to finish bootstrapping
I20260808 07:03:09.190241 16362 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 90a2711b680e473d9a1f95824ff5a118 to finish bootstrapping
I20260808 07:03:09.197281 22972 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ddcb98a444444aea9ea6e92101a89bbd"
dest_uuid: "82f384a95529443683b2870ca150ce97"
from {username='slave'} at 127.0.0.1:35564
I20260808 07:03:09.197399 22972 raft_consensus.cc:482] T ddcb98a444444aea9ea6e92101a89bbd P 82f384a95529443683b2870ca150ce97 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 07:03:09.197440 22972 raft_consensus.cc:3037] T ddcb98a444444aea9ea6e92101a89bbd P 82f384a95529443683b2870ca150ce97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:09.198137 22972 raft_consensus.cc:504] T ddcb98a444444aea9ea6e92101a89bbd P 82f384a95529443683b2870ca150ce97 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f384a95529443683b2870ca150ce97" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34447 } } peers { permanent_uuid: "90a2711b680e473d9a1f95824ff5a118" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39363 } } peers { permanent_uuid: "4c604fa7ff214d748eb788f7b7a9f695" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 40993 } }
I20260808 07:03:09.198426 22972 leader_election.cc:288] T ddcb98a444444aea9ea6e92101a89bbd P 82f384a95529443683b2870ca150ce97 [CANDIDATE]: Term 1 election: Requested vote from peers 90a2711b680e473d9a1f95824ff5a118 (127.15.250.131:39363), 4c604fa7ff214d748eb788f7b7a9f695 (127.15.250.129:40993)
I20260808 07:03:09.201637 23109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddcb98a444444aea9ea6e92101a89bbd" candidate_uuid: "82f384a95529443683b2870ca150ce97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "90a2711b680e473d9a1f95824ff5a118"
I20260808 07:03:09.201746 23109 raft_consensus.cc:3037] T ddcb98a444444aea9ea6e92101a89bbd P 90a2711b680e473d9a1f95824ff5a118 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:09.202509 23109 raft_consensus.cc:2454] T ddcb98a444444aea9ea6e92101a89bbd P 90a2711b680e473d9a1f95824ff5a118 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82f384a95529443683b2870ca150ce97 in term 1.
I20260808 07:03:09.202800 22903 leader_election.cc:302] T ddcb98a444444aea9ea6e92101a89bbd P 82f384a95529443683b2870ca150ce97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82f384a95529443683b2870ca150ce97, 90a2711b680e473d9a1f95824ff5a118; no voters:
I20260808 07:03:09.202970 23181 raft_consensus.cc:2781] T ddcb98a444444aea9ea6e92101a89bbd P 82f384a95529443683b2870ca150ce97 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:09.202965 22835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddcb98a444444aea9ea6e92101a89bbd" candidate_uuid: "82f384a95529443683b2870ca150ce97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4c604fa7ff214d748eb788f7b7a9f695"
I20260808 07:03:09.203136 23181 raft_consensus.cc:686] T ddcb98a444444aea9ea6e92101a89bbd P 82f384a95529443683b2870ca150ce97 [term 1 LEADER]: Becoming Leader. State: Replica: 82f384a95529443683b2870ca150ce97, State: Running, Role: LEADER
I20260808 07:03:09.203096 22835 raft_consensus.cc:3037] T ddcb98a444444aea9ea6e92101a89bbd P 4c604fa7ff214d748eb788f7b7a9f695 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:09.203236 23181 consensus_queue.cc:237] T ddcb98a444444aea9ea6e92101a89bbd P 82f384a95529443683b2870ca150ce97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f384a95529443683b2870ca150ce97" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34447 } } peers { permanent_uuid: "90a2711b680e473d9a1f95824ff5a118" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39363 } } peers { permanent_uuid: "4c604fa7ff214d748eb788f7b7a9f695" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 40993 } }
I20260808 07:03:09.204108 22835 raft_consensus.cc:2454] T ddcb98a444444aea9ea6e92101a89bbd P 4c604fa7ff214d748eb788f7b7a9f695 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82f384a95529443683b2870ca150ce97 in term 1.
I20260808 07:03:09.204207 22687 catalog_manager.cc:5495] T ddcb98a444444aea9ea6e92101a89bbd P 82f384a95529443683b2870ca150ce97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82f384a95529443683b2870ca150ce97 (127.15.250.130). New cstate: current_term: 1 leader_uuid: "82f384a95529443683b2870ca150ce97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f384a95529443683b2870ca150ce97" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a2711b680e473d9a1f95824ff5a118" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c604fa7ff214d748eb788f7b7a9f695" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 40993 } health_report { overall_health: UNKNOWN } } }
W20260808 07:03:09.248154 23023 tablet.cc:2369] T ddcb98a444444aea9ea6e92101a89bbd P 82f384a95529443683b2870ca150ce97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 07:03:10.225818 22972 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ddcb98a444444aea9ea6e92101a89bbd"
dest_uuid: "82f384a95529443683b2870ca150ce97"
from {username='slave'} at 127.0.0.1:35564
I20260808 07:03:10.225987 22972 raft_consensus.cc:580] T ddcb98a444444aea9ea6e92101a89bbd P 82f384a95529443683b2870ca150ce97 [term 1 LEADER]: Received request to step down
I20260808 07:03:10.226019 22972 raft_consensus.cc:3032] T ddcb98a444444aea9ea6e92101a89bbd P 82f384a95529443683b2870ca150ce97 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 07:03:10.226050 22972 raft_consensus.cc:729] T ddcb98a444444aea9ea6e92101a89bbd P 82f384a95529443683b2870ca150ce97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 82f384a95529443683b2870ca150ce97, State: Running, Role: LEADER
I20260808 07:03:10.226104 22972 consensus_queue.cc:260] T ddcb98a444444aea9ea6e92101a89bbd P 82f384a95529443683b2870ca150ce97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f384a95529443683b2870ca150ce97" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34447 } } peers { permanent_uuid: "90a2711b680e473d9a1f95824ff5a118" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39363 } } peers { permanent_uuid: "4c604fa7ff214d748eb788f7b7a9f695" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 40993 } }
I20260808 07:03:10.226250 22972 raft_consensus.cc:3037] T ddcb98a444444aea9ea6e92101a89bbd P 82f384a95529443683b2870ca150ce97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 07:03:10.226682 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 22749
W20260808 07:03:10.232669 22903 connection.cc:535] client connection to 127.15.250.129:40993 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 07:03:10.232817 22903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 07:03:10.233158 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 22886
I20260808 07:03:10.238406 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 23025
I20260808 07:03:10.243643 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 22654
2026-08-08T07:03:10Z chronyd exiting
[ OK ] RaftConsensusElectionITest.StepDownWithSlowFollower (1692 ms)
[ RUN ] RaftConsensusElectionITest.ElectPendingVoter
I20260808 07:03:10.264933 16362 test_util.cc:268] Using random seed: -1228926368
I20260808 07:03:10.265476 16362 ts_itest-base.cc:112] Starting cluster with:
I20260808 07:03:10.265522 16362 ts_itest-base.cc:113] --------------
I20260808 07:03:10.265533 16362 ts_itest-base.cc:114] 5 tablet servers
I20260808 07:03:10.265543 16362 ts_itest-base.cc:115] 5 replicas per TS
I20260808 07:03:10.265554 16362 ts_itest-base.cc:116] --------------
2026-08-08T07:03:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:03:10Z Disabled control of system clock
I20260808 07:03:10.274883 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.250.190:34879
--webserver_interface=127.15.250.190
--webserver_port=0
--builtin_ntp_servers=127.15.250.148:44517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.250.190:34879
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20260808 07:03:10.356314 23194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:10.356563 23194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:10.356595 23194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:10.357781 23194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:03:10.357848 23194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:10.357874 23194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:03:10.357908 23194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:03:10.359310 23194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:44517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.15.250.190:34879
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.250.190:34879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.250.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.23194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:10.359586 23194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:10.359876 23194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:10.364871 23199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:10.364884 23202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:10.364915 23194 server_base.cc:1034] running on GCE node
W20260808 07:03:10.364871 23200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:10.365356 23194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:10.365662 23194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:10.366842 23194 hybrid_clock.cc:648] HybridClock initialized: now 1786172590366823 us; error 32 us; skew 500 ppm
I20260808 07:03:10.368090 23194 webserver.cc:460] Webserver started at http://127.15.250.190:35875/ using document root <none> and password file <none>
I20260808 07:03:10.368431 23194 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:10.368512 23194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:10.368654 23194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:10.369738 23194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "c8765dba6127477389139343d9057e0b"
format_stamp: "Formatted at 2026-08-08 07:03:10 on dist-test-slave-0b58"
I20260808 07:03:10.370118 23194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "c8765dba6127477389139343d9057e0b"
format_stamp: "Formatted at 2026-08-08 07:03:10 on dist-test-slave-0b58"
I20260808 07:03:10.371433 23194 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 07:03:10.372157 23208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:10.372378 23194 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:03:10.372481 23194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
uuid: "c8765dba6127477389139343d9057e0b"
format_stamp: "Formatted at 2026-08-08 07:03:10 on dist-test-slave-0b58"
I20260808 07:03:10.372567 23194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:10.399248 23194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:10.399642 23194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:10.399827 23194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:10.403919 23194 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.190:34879
I20260808 07:03:10.403982 23265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.190:34879 every 8 connection(s)
I20260808 07:03:10.404412 23194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260808 07:03:10.405117 23266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:10.407636 23266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8765dba6127477389139343d9057e0b: Bootstrap starting.
I20260808 07:03:10.408296 23266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8765dba6127477389139343d9057e0b: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:10.408603 23266 log.cc:824] T 00000000000000000000000000000000 P c8765dba6127477389139343d9057e0b: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:10.409300 23266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8765dba6127477389139343d9057e0b: No bootstrap required, opened a new log
I20260808 07:03:10.409897 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 23194
I20260808 07:03:10.410023 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260808 07:03:10.411443 23266 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8765dba6127477389139343d9057e0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8765dba6127477389139343d9057e0b" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 34879 } }
I20260808 07:03:10.411619 23266 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8765dba6127477389139343d9057e0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:03:10.411659 23266 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8765dba6127477389139343d9057e0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8765dba6127477389139343d9057e0b, State: Initialized, Role: FOLLOWER
I20260808 07:03:10.411832 23266 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8765dba6127477389139343d9057e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8765dba6127477389139343d9057e0b" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 34879 } }
I20260808 07:03:10.411923 23266 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8765dba6127477389139343d9057e0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:03:10.411959 23266 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8765dba6127477389139343d9057e0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:03:10.412011 23266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8765dba6127477389139343d9057e0b [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:10.412774 23266 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8765dba6127477389139343d9057e0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8765dba6127477389139343d9057e0b" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 34879 } }
I20260808 07:03:10.412884 23266 leader_election.cc:302] T 00000000000000000000000000000000 P c8765dba6127477389139343d9057e0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8765dba6127477389139343d9057e0b; no voters:
I20260808 07:03:10.413028 23266 leader_election.cc:288] T 00000000000000000000000000000000 P c8765dba6127477389139343d9057e0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:03:10.413100 23271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8765dba6127477389139343d9057e0b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:10.413272 23266 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8765dba6127477389139343d9057e0b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:03:10.413291 23271 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8765dba6127477389139343d9057e0b [term 1 LEADER]: Becoming Leader. State: Replica: c8765dba6127477389139343d9057e0b, State: Running, Role: LEADER
I20260808 07:03:10.413544 23271 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8765dba6127477389139343d9057e0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8765dba6127477389139343d9057e0b" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 34879 } }
I20260808 07:03:10.414023 23272 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8765dba6127477389139343d9057e0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8765dba6127477389139343d9057e0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8765dba6127477389139343d9057e0b" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 34879 } } }
I20260808 07:03:10.414099 23272 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8765dba6127477389139343d9057e0b [sys.catalog]: This master's current role is: LEADER
I20260808 07:03:10.415323 23273 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8765dba6127477389139343d9057e0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8765dba6127477389139343d9057e0b. Latest consensus state: current_term: 1 leader_uuid: "c8765dba6127477389139343d9057e0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8765dba6127477389139343d9057e0b" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 34879 } } }
I20260808 07:03:10.415408 23273 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8765dba6127477389139343d9057e0b [sys.catalog]: This master's current role is: LEADER
I20260808 07:03:10.415591 23287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:03:10.416157 23287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:03:10.418026 23287 catalog_manager.cc:1347] Generated new cluster ID: ba10d3be9e6e4c16aad5f0699d02954e
I20260808 07:03:10.418116 23287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:03:10.426000 23287 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:03:10.426520 23287 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:03:10.432278 23287 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8765dba6127477389139343d9057e0b: Generated new TSK 0
I20260808 07:03:10.432480 23287 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:03:10.438232 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.129:0
--local_ip_for_outbound_sockets=127.15.250.129
--webserver_interface=127.15.250.129
--webserver_port=0
--tserver_master_addrs=127.15.250.190:34879
--builtin_ntp_servers=127.15.250.148:44517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20260808 07:03:10.521636 23291 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 07:03:10.521894 23291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:10.521922 23291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:10.521953 23291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:10.523155 23291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:10.523243 23291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.129
I20260808 07:03:10.524734 23291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:44517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.250.129
--webserver_port=0
--tserver_master_addrs=127.15.250.190:34879
--never_fsync=true
--heap_profile_path=/tmp/kudu.23291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.129
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:10.525002 23291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:10.525305 23291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:10.530870 23299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:10.530871 23296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:10.531057 23291 server_base.cc:1034] running on GCE node
W20260808 07:03:10.531292 23297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:10.531558 23291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:10.531818 23291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:10.532990 23291 hybrid_clock.cc:648] HybridClock initialized: now 1786172590532975 us; error 31 us; skew 500 ppm
I20260808 07:03:10.534300 23291 webserver.cc:460] Webserver started at http://127.15.250.129:33695/ using document root <none> and password file <none>
I20260808 07:03:10.534550 23291 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:10.534619 23291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:10.534759 23291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:10.535797 23291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "cf70a94b41d24bcab02a0e2beebac61d"
format_stamp: "Formatted at 2026-08-08 07:03:10 on dist-test-slave-0b58"
I20260808 07:03:10.536157 23291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "cf70a94b41d24bcab02a0e2beebac61d"
format_stamp: "Formatted at 2026-08-08 07:03:10 on dist-test-slave-0b58"
I20260808 07:03:10.537680 23291 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 07:03:10.538605 23305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:10.538813 23291 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 07:03:10.538919 23291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "cf70a94b41d24bcab02a0e2beebac61d"
format_stamp: "Formatted at 2026-08-08 07:03:10 on dist-test-slave-0b58"
I20260808 07:03:10.539003 23291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:10.554908 23291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:10.555236 23291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:10.555392 23291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:10.555655 23291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:03:10.555999 23291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:03:10.556034 23291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:10.556058 23291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:03:10.556104 23291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:10.562166 23291 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.129:45771
I20260808 07:03:10.562208 23421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.129:45771 every 8 connection(s)
I20260808 07:03:10.562556 23291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260808 07:03:10.564086 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 23291
I20260808 07:03:10.564195 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260808 07:03:10.565625 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.130:0
--local_ip_for_outbound_sockets=127.15.250.130
--webserver_interface=127.15.250.130
--webserver_port=0
--tserver_master_addrs=127.15.250.190:34879
--builtin_ntp_servers=127.15.250.148:44517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260808 07:03:10.569824 23422 heartbeater.cc:347] Connected to a master server at 127.15.250.190:34879
I20260808 07:03:10.569923 23422 heartbeater.cc:464] Registering TS with master...
I20260808 07:03:10.570111 23422 heartbeater.cc:511] Master 127.15.250.190:34879 requested a full tablet report, sending...
I20260808 07:03:10.570631 23227 ts_manager.cc:194] Registered new tserver with Master: cf70a94b41d24bcab02a0e2beebac61d (127.15.250.129:45771)
I20260808 07:03:10.571250 23227 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.129:51289
W20260808 07:03:10.648800 23425 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 07:03:10.649065 23425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:10.649085 23425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:10.649109 23425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:10.650259 23425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:10.650314 23425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.130
I20260808 07:03:10.651710 23425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:44517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.250.130
--webserver_port=0
--tserver_master_addrs=127.15.250.190:34879
--never_fsync=true
--heap_profile_path=/tmp/kudu.23425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.130
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:10.651912 23425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:10.652148 23425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:10.657691 23431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:10.657682 23430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:10.657721 23425 server_base.cc:1034] running on GCE node
W20260808 07:03:10.657684 23433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:10.658074 23425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:10.658344 23425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:10.659499 23425 hybrid_clock.cc:648] HybridClock initialized: now 1786172590659474 us; error 40 us; skew 500 ppm
I20260808 07:03:10.660848 23425 webserver.cc:460] Webserver started at http://127.15.250.130:45657/ using document root <none> and password file <none>
I20260808 07:03:10.661046 23425 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:10.661087 23425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:10.661182 23425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:10.662169 23425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "9e33cc47976341d8898174a800a08ff5"
format_stamp: "Formatted at 2026-08-08 07:03:10 on dist-test-slave-0b58"
I20260808 07:03:10.662474 23425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "9e33cc47976341d8898174a800a08ff5"
format_stamp: "Formatted at 2026-08-08 07:03:10 on dist-test-slave-0b58"
I20260808 07:03:10.663729 23425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 07:03:10.664436 23439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:10.664614 23425 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 07:03:10.664719 23425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "9e33cc47976341d8898174a800a08ff5"
format_stamp: "Formatted at 2026-08-08 07:03:10 on dist-test-slave-0b58"
I20260808 07:03:10.664775 23425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:10.682291 23425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:10.682561 23425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:10.682670 23425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:10.682952 23425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:03:10.683297 23425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:03:10.683354 23425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:10.683398 23425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:03:10.683425 23425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:10.689132 23425 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.130:41633
I20260808 07:03:10.689172 23556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.130:41633 every 8 connection(s)
I20260808 07:03:10.689559 23425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260808 07:03:10.691286 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 23425
I20260808 07:03:10.691397 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260808 07:03:10.692555 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.131:0
--local_ip_for_outbound_sockets=127.15.250.131
--webserver_interface=127.15.250.131
--webserver_port=0
--tserver_master_addrs=127.15.250.190:34879
--builtin_ntp_servers=127.15.250.148:44517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260808 07:03:10.696038 23557 heartbeater.cc:347] Connected to a master server at 127.15.250.190:34879
I20260808 07:03:10.696130 23557 heartbeater.cc:464] Registering TS with master...
I20260808 07:03:10.696379 23557 heartbeater.cc:511] Master 127.15.250.190:34879 requested a full tablet report, sending...
I20260808 07:03:10.696789 23227 ts_manager.cc:194] Registered new tserver with Master: 9e33cc47976341d8898174a800a08ff5 (127.15.250.130:41633)
I20260808 07:03:10.697345 23227 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.130:42863
W20260808 07:03:10.776366 23560 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 07:03:10.776571 23560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:10.776587 23560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:10.776604 23560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:10.777801 23560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:10.777859 23560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.131
I20260808 07:03:10.779210 23560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:44517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.250.131
--webserver_port=0
--tserver_master_addrs=127.15.250.190:34879
--never_fsync=true
--heap_profile_path=/tmp/kudu.23560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.131
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:10.779475 23560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:10.779795 23560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:10.785126 23568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:10.785127 23566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:10.785221 23560 server_base.cc:1034] running on GCE node
W20260808 07:03:10.785236 23565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:10.785585 23560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:10.785827 23560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:10.786994 23560 hybrid_clock.cc:648] HybridClock initialized: now 1786172590786976 us; error 38 us; skew 500 ppm
I20260808 07:03:10.788426 23560 webserver.cc:460] Webserver started at http://127.15.250.131:33483/ using document root <none> and password file <none>
I20260808 07:03:10.788700 23560 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:10.788770 23560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:10.788931 23560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:10.789916 23560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "eec3c50d1e4043c28a9ec95776302146"
format_stamp: "Formatted at 2026-08-08 07:03:10 on dist-test-slave-0b58"
I20260808 07:03:10.790272 23560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "eec3c50d1e4043c28a9ec95776302146"
format_stamp: "Formatted at 2026-08-08 07:03:10 on dist-test-slave-0b58"
I20260808 07:03:10.791661 23560 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 07:03:10.792461 23574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:10.792752 23560 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:03:10.792865 23560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "eec3c50d1e4043c28a9ec95776302146"
format_stamp: "Formatted at 2026-08-08 07:03:10 on dist-test-slave-0b58"
I20260808 07:03:10.792941 23560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:10.803264 23560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:10.803573 23560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:10.803722 23560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:10.803946 23560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:03:10.804342 23560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:03:10.804395 23560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:10.804436 23560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:03:10.804462 23560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:10.810355 23560 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.131:35751
I20260808 07:03:10.810411 23692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.131:35751 every 8 connection(s)
I20260808 07:03:10.810727 23560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260808 07:03:10.816727 23693 heartbeater.cc:347] Connected to a master server at 127.15.250.190:34879
I20260808 07:03:10.816856 23693 heartbeater.cc:464] Registering TS with master...
I20260808 07:03:10.817050 23693 heartbeater.cc:511] Master 127.15.250.190:34879 requested a full tablet report, sending...
I20260808 07:03:10.817564 23227 ts_manager.cc:194] Registered new tserver with Master: eec3c50d1e4043c28a9ec95776302146 (127.15.250.131:35751)
I20260808 07:03:10.817971 23227 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.131:40499
I20260808 07:03:10.818959 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 23560
I20260808 07:03:10.819080 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260808 07:03:10.820402 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.132:0
--local_ip_for_outbound_sockets=127.15.250.132
--webserver_interface=127.15.250.132
--webserver_port=0
--tserver_master_addrs=127.15.250.190:34879
--builtin_ntp_servers=127.15.250.148:44517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20260808 07:03:10.905210 23696 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 07:03:10.905512 23696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:10.905532 23696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:10.905550 23696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:10.906670 23696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:10.906716 23696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.132
I20260808 07:03:10.907999 23696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:44517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.15.250.132
--webserver_port=0
--tserver_master_addrs=127.15.250.190:34879
--never_fsync=true
--heap_profile_path=/tmp/kudu.23696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.132
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:10.908197 23696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:10.908545 23696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:10.914147 23702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:10.914134 23704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:10.914327 23696 server_base.cc:1034] running on GCE node
W20260808 07:03:10.914441 23701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:10.914690 23696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:10.914908 23696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:10.916112 23696 hybrid_clock.cc:648] HybridClock initialized: now 1786172590916094 us; error 50 us; skew 500 ppm
I20260808 07:03:10.917502 23696 webserver.cc:460] Webserver started at http://127.15.250.132:43489/ using document root <none> and password file <none>
I20260808 07:03:10.917817 23696 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:10.917879 23696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:10.917975 23696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:10.918984 23696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "9ad636a04bcf4998bf21c4126858d557"
format_stamp: "Formatted at 2026-08-08 07:03:10 on dist-test-slave-0b58"
I20260808 07:03:10.919294 23696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "9ad636a04bcf4998bf21c4126858d557"
format_stamp: "Formatted at 2026-08-08 07:03:10 on dist-test-slave-0b58"
I20260808 07:03:10.920600 23696 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 07:03:10.921273 23711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:10.921440 23696 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:10.921541 23696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "9ad636a04bcf4998bf21c4126858d557"
format_stamp: "Formatted at 2026-08-08 07:03:10 on dist-test-slave-0b58"
I20260808 07:03:10.921631 23696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:10.945287 23696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:10.945618 23696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:10.945775 23696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:10.946070 23696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:03:10.946455 23696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:03:10.946511 23696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:10.946555 23696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:03:10.946581 23696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:10.952471 23696 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.132:41747
I20260808 07:03:10.952527 23828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.132:41747 every 8 connection(s)
I20260808 07:03:10.952878 23696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20260808 07:03:10.956198 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 23696
I20260808 07:03:10.956341 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20260808 07:03:10.957512 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.133:0
--local_ip_for_outbound_sockets=127.15.250.133
--webserver_interface=127.15.250.133
--webserver_port=0
--tserver_master_addrs=127.15.250.190:34879
--builtin_ntp_servers=127.15.250.148:44517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260808 07:03:10.958966 23829 heartbeater.cc:347] Connected to a master server at 127.15.250.190:34879
I20260808 07:03:10.959076 23829 heartbeater.cc:464] Registering TS with master...
I20260808 07:03:10.959303 23829 heartbeater.cc:511] Master 127.15.250.190:34879 requested a full tablet report, sending...
I20260808 07:03:10.959710 23227 ts_manager.cc:194] Registered new tserver with Master: 9ad636a04bcf4998bf21c4126858d557 (127.15.250.132:41747)
I20260808 07:03:10.960100 23227 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.132:42805
W20260808 07:03:11.040517 23832 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 07:03:11.040763 23832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:11.040791 23832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:11.040830 23832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:11.041965 23832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:11.042044 23832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.133
I20260808 07:03:11.043383 23832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:44517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.15.250.133
--webserver_port=0
--tserver_master_addrs=127.15.250.190:34879
--never_fsync=true
--heap_profile_path=/tmp/kudu.23832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.133
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:11.043756 23832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:11.044083 23832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:11.049214 23841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:11.049214 23838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:11.049391 23839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:11.049567 23832 server_base.cc:1034] running on GCE node
I20260808 07:03:11.049937 23832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:11.050213 23832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:11.051389 23832 hybrid_clock.cc:648] HybridClock initialized: now 1786172591051372 us; error 32 us; skew 500 ppm
I20260808 07:03:11.052973 23832 webserver.cc:460] Webserver started at http://127.15.250.133:34737/ using document root <none> and password file <none>
I20260808 07:03:11.053270 23832 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:11.053331 23832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:11.053431 23832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:11.054387 23832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "e520d4b30f204741997deaf911059ce6"
format_stamp: "Formatted at 2026-08-08 07:03:11 on dist-test-slave-0b58"
I20260808 07:03:11.054701 23832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "e520d4b30f204741997deaf911059ce6"
format_stamp: "Formatted at 2026-08-08 07:03:11 on dist-test-slave-0b58"
I20260808 07:03:11.056119 23832 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 07:03:11.056969 23848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:11.057161 23832 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:03:11.057250 23832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "e520d4b30f204741997deaf911059ce6"
format_stamp: "Formatted at 2026-08-08 07:03:11 on dist-test-slave-0b58"
I20260808 07:03:11.057309 23832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:11.074892 23832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:11.075186 23832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:11.075289 23832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:11.075536 23832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:03:11.075901 23832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:03:11.075934 23832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:11.075958 23832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:03:11.075985 23832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:11.081758 23832 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.133:45281
I20260808 07:03:11.081825 23966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.133:45281 every 8 connection(s)
I20260808 07:03:11.082177 23832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20260808 07:03:11.083789 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 23832
I20260808 07:03:11.083900 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20260808 07:03:11.088166 23967 heartbeater.cc:347] Connected to a master server at 127.15.250.190:34879
I20260808 07:03:11.088328 23967 heartbeater.cc:464] Registering TS with master...
I20260808 07:03:11.088515 23967 heartbeater.cc:511] Master 127.15.250.190:34879 requested a full tablet report, sending...
I20260808 07:03:11.088872 23227 ts_manager.cc:194] Registered new tserver with Master: e520d4b30f204741997deaf911059ce6 (127.15.250.133:45281)
I20260808 07:03:11.089236 23227 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.133:33959
I20260808 07:03:11.095919 16362 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20260808 07:03:11.102228 23227 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43612:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20260808 07:03:11.103845 23227 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 6 tablet servers would be needed, 5 are available. Consider bringing up more tablet servers.
I20260808 07:03:11.112140 23489 tablet_service.cc:1467] Processing CreateTablet for tablet 3e8975b6bb314806a9b5afdc9a22253a (DEFAULT_TABLE table=TestTable [id=2d4d5a414a4446eaa1f63b694c062d8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:03:11.112588 23489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e8975b6bb314806a9b5afdc9a22253a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:11.112588 23626 tablet_service.cc:1467] Processing CreateTablet for tablet 3e8975b6bb314806a9b5afdc9a22253a (DEFAULT_TABLE table=TestTable [id=2d4d5a414a4446eaa1f63b694c062d8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:03:11.112954 23626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e8975b6bb314806a9b5afdc9a22253a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:11.113184 23759 tablet_service.cc:1467] Processing CreateTablet for tablet 3e8975b6bb314806a9b5afdc9a22253a (DEFAULT_TABLE table=TestTable [id=2d4d5a414a4446eaa1f63b694c062d8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:03:11.113570 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e8975b6bb314806a9b5afdc9a22253a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:11.116003 23990 tablet_bootstrap.cc:492] T 3e8975b6bb314806a9b5afdc9a22253a P eec3c50d1e4043c28a9ec95776302146: Bootstrap starting.
I20260808 07:03:11.116807 23990 tablet_bootstrap.cc:654] T 3e8975b6bb314806a9b5afdc9a22253a P eec3c50d1e4043c28a9ec95776302146: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:11.116731 23991 tablet_bootstrap.cc:492] T 3e8975b6bb314806a9b5afdc9a22253a P 9ad636a04bcf4998bf21c4126858d557: Bootstrap starting.
I20260808 07:03:11.117228 23897 tablet_service.cc:1467] Processing CreateTablet for tablet 3e8975b6bb314806a9b5afdc9a22253a (DEFAULT_TABLE table=TestTable [id=2d4d5a414a4446eaa1f63b694c062d8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:03:11.117560 23991 tablet_bootstrap.cc:654] T 3e8975b6bb314806a9b5afdc9a22253a P 9ad636a04bcf4998bf21c4126858d557: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:11.117578 23897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e8975b6bb314806a9b5afdc9a22253a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:11.117713 23990 log.cc:824] T 3e8975b6bb314806a9b5afdc9a22253a P eec3c50d1e4043c28a9ec95776302146: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:11.117909 23991 log.cc:824] T 3e8975b6bb314806a9b5afdc9a22253a P 9ad636a04bcf4998bf21c4126858d557: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:11.118187 23989 tablet_bootstrap.cc:492] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5: Bootstrap starting.
I20260808 07:03:11.118435 23354 tablet_service.cc:1467] Processing CreateTablet for tablet 3e8975b6bb314806a9b5afdc9a22253a (DEFAULT_TABLE table=TestTable [id=2d4d5a414a4446eaa1f63b694c062d8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:03:11.118768 23354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e8975b6bb314806a9b5afdc9a22253a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:11.118861 23989 tablet_bootstrap.cc:654] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:11.119175 23989 log.cc:824] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:11.119942 23989 tablet_bootstrap.cc:492] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5: No bootstrap required, opened a new log
I20260808 07:03:11.120018 23989 ts_tablet_manager.cc:1397] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:03:11.120788 23990 tablet_bootstrap.cc:492] T 3e8975b6bb314806a9b5afdc9a22253a P eec3c50d1e4043c28a9ec95776302146: No bootstrap required, opened a new log
I20260808 07:03:11.120934 23990 ts_tablet_manager.cc:1397] T 3e8975b6bb314806a9b5afdc9a22253a P eec3c50d1e4043c28a9ec95776302146: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260808 07:03:11.121068 23995 tablet_bootstrap.cc:492] T 3e8975b6bb314806a9b5afdc9a22253a P e520d4b30f204741997deaf911059ce6: Bootstrap starting.
I20260808 07:03:11.121788 23995 tablet_bootstrap.cc:654] T 3e8975b6bb314806a9b5afdc9a22253a P e520d4b30f204741997deaf911059ce6: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:11.122136 23995 log.cc:824] T 3e8975b6bb314806a9b5afdc9a22253a P e520d4b30f204741997deaf911059ce6: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:11.122262 23996 tablet_bootstrap.cc:492] T 3e8975b6bb314806a9b5afdc9a22253a P cf70a94b41d24bcab02a0e2beebac61d: Bootstrap starting.
I20260808 07:03:11.122788 23995 tablet_bootstrap.cc:492] T 3e8975b6bb314806a9b5afdc9a22253a P e520d4b30f204741997deaf911059ce6: No bootstrap required, opened a new log
I20260808 07:03:11.122859 23995 ts_tablet_manager.cc:1397] T 3e8975b6bb314806a9b5afdc9a22253a P e520d4b30f204741997deaf911059ce6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:03:11.122992 23996 tablet_bootstrap.cc:654] T 3e8975b6bb314806a9b5afdc9a22253a P cf70a94b41d24bcab02a0e2beebac61d: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:11.123313 23996 log.cc:824] T 3e8975b6bb314806a9b5afdc9a22253a P cf70a94b41d24bcab02a0e2beebac61d: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:11.123389 23989 raft_consensus.cc:348] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } } peers { permanent_uuid: "eec3c50d1e4043c28a9ec95776302146" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 35751 } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } }
I20260808 07:03:11.123603 23989 raft_consensus.cc:729] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e33cc47976341d8898174a800a08ff5, State: Initialized, Role: FOLLOWER
I20260808 07:03:11.123919 23989 consensus_queue.cc:260] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } } peers { permanent_uuid: "eec3c50d1e4043c28a9ec95776302146" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 35751 } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } }
I20260808 07:03:11.124573 23989 ts_tablet_manager.cc:1428] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260808 07:03:11.124758 23557 heartbeater.cc:503] Master 127.15.250.190:34879 was elected leader, sending a full tablet report...
I20260808 07:03:11.124886 23990 raft_consensus.cc:348] T 3e8975b6bb314806a9b5afdc9a22253a P eec3c50d1e4043c28a9ec95776302146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } } peers { permanent_uuid: "eec3c50d1e4043c28a9ec95776302146" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 35751 } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } }
I20260808 07:03:11.125129 23990 raft_consensus.cc:729] T 3e8975b6bb314806a9b5afdc9a22253a P eec3c50d1e4043c28a9ec95776302146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eec3c50d1e4043c28a9ec95776302146, State: Initialized, Role: FOLLOWER
I20260808 07:03:11.125288 23990 consensus_queue.cc:260] T 3e8975b6bb314806a9b5afdc9a22253a P eec3c50d1e4043c28a9ec95776302146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } } peers { permanent_uuid: "eec3c50d1e4043c28a9ec95776302146" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 35751 } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } }
I20260808 07:03:11.125675 23990 ts_tablet_manager.cc:1428] T 3e8975b6bb314806a9b5afdc9a22253a P eec3c50d1e4043c28a9ec95776302146: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260808 07:03:11.125829 23693 heartbeater.cc:503] Master 127.15.250.190:34879 was elected leader, sending a full tablet report...
I20260808 07:03:11.126026 23995 raft_consensus.cc:348] T 3e8975b6bb314806a9b5afdc9a22253a P e520d4b30f204741997deaf911059ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } } peers { permanent_uuid: "eec3c50d1e4043c28a9ec95776302146" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 35751 } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } }
I20260808 07:03:11.126297 23995 raft_consensus.cc:729] T 3e8975b6bb314806a9b5afdc9a22253a P e520d4b30f204741997deaf911059ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e520d4b30f204741997deaf911059ce6, State: Initialized, Role: FOLLOWER
I20260808 07:03:11.126518 23995 consensus_queue.cc:260] T 3e8975b6bb314806a9b5afdc9a22253a P e520d4b30f204741997deaf911059ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } } peers { permanent_uuid: "eec3c50d1e4043c28a9ec95776302146" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 35751 } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } }
I20260808 07:03:11.126832 23995 ts_tablet_manager.cc:1428] T 3e8975b6bb314806a9b5afdc9a22253a P e520d4b30f204741997deaf911059ce6: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260808 07:03:11.127606 23967 heartbeater.cc:503] Master 127.15.250.190:34879 was elected leader, sending a full tablet report...
I20260808 07:03:11.128579 23991 tablet_bootstrap.cc:492] T 3e8975b6bb314806a9b5afdc9a22253a P 9ad636a04bcf4998bf21c4126858d557: No bootstrap required, opened a new log
I20260808 07:03:11.128662 23991 ts_tablet_manager.cc:1397] T 3e8975b6bb314806a9b5afdc9a22253a P 9ad636a04bcf4998bf21c4126858d557: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20260808 07:03:11.129146 16362 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20260808 07:03:11.129657 16362 ts_itest-base.cc:233] Waiting for 1 tablets on tserver cf70a94b41d24bcab02a0e2beebac61d to finish bootstrapping
I20260808 07:03:11.131130 23991 raft_consensus.cc:348] T 3e8975b6bb314806a9b5afdc9a22253a P 9ad636a04bcf4998bf21c4126858d557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } } peers { permanent_uuid: "eec3c50d1e4043c28a9ec95776302146" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 35751 } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } }
I20260808 07:03:11.131309 23991 raft_consensus.cc:729] T 3e8975b6bb314806a9b5afdc9a22253a P 9ad636a04bcf4998bf21c4126858d557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ad636a04bcf4998bf21c4126858d557, State: Initialized, Role: FOLLOWER
I20260808 07:03:11.131395 23991 consensus_queue.cc:260] T 3e8975b6bb314806a9b5afdc9a22253a P 9ad636a04bcf4998bf21c4126858d557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } } peers { permanent_uuid: "eec3c50d1e4043c28a9ec95776302146" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 35751 } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } }
I20260808 07:03:11.131691 23991 ts_tablet_manager.cc:1428] T 3e8975b6bb314806a9b5afdc9a22253a P 9ad636a04bcf4998bf21c4126858d557: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 07:03:11.131738 23829 heartbeater.cc:503] Master 127.15.250.190:34879 was elected leader, sending a full tablet report...
I20260808 07:03:11.132496 23996 tablet_bootstrap.cc:492] T 3e8975b6bb314806a9b5afdc9a22253a P cf70a94b41d24bcab02a0e2beebac61d: No bootstrap required, opened a new log
I20260808 07:03:11.132650 23996 ts_tablet_manager.cc:1397] T 3e8975b6bb314806a9b5afdc9a22253a P cf70a94b41d24bcab02a0e2beebac61d: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260808 07:03:11.134972 23996 raft_consensus.cc:348] T 3e8975b6bb314806a9b5afdc9a22253a P cf70a94b41d24bcab02a0e2beebac61d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } } peers { permanent_uuid: "eec3c50d1e4043c28a9ec95776302146" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 35751 } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } }
I20260808 07:03:11.135146 23996 raft_consensus.cc:729] T 3e8975b6bb314806a9b5afdc9a22253a P cf70a94b41d24bcab02a0e2beebac61d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf70a94b41d24bcab02a0e2beebac61d, State: Initialized, Role: FOLLOWER
I20260808 07:03:11.135263 23996 consensus_queue.cc:260] T 3e8975b6bb314806a9b5afdc9a22253a P cf70a94b41d24bcab02a0e2beebac61d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } } peers { permanent_uuid: "eec3c50d1e4043c28a9ec95776302146" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 35751 } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } }
I20260808 07:03:11.135545 23996 ts_tablet_manager.cc:1428] T 3e8975b6bb314806a9b5afdc9a22253a P cf70a94b41d24bcab02a0e2beebac61d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 07:03:11.135607 23422 heartbeater.cc:503] Master 127.15.250.190:34879 was elected leader, sending a full tablet report...
I20260808 07:03:11.145040 16362 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 9e33cc47976341d8898174a800a08ff5 to finish bootstrapping
I20260808 07:03:11.151911 16362 ts_itest-base.cc:233] Waiting for 1 tablets on tserver eec3c50d1e4043c28a9ec95776302146 to finish bootstrapping
I20260808 07:03:11.156421 16362 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 9ad636a04bcf4998bf21c4126858d557 to finish bootstrapping
I20260808 07:03:11.160714 16362 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e520d4b30f204741997deaf911059ce6 to finish bootstrapping
I20260808 07:03:11.167618 23510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3e8975b6bb314806a9b5afdc9a22253a"
dest_uuid: "9e33cc47976341d8898174a800a08ff5"
from {username='slave'} at 127.0.0.1:34020
I20260808 07:03:11.167739 23510 raft_consensus.cc:482] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 07:03:11.167783 23510 raft_consensus.cc:3037] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:11.168653 23510 raft_consensus.cc:504] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } } peers { permanent_uuid: "eec3c50d1e4043c28a9ec95776302146" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 35751 } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } }
I20260808 07:03:11.169031 23510 leader_election.cc:288] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5 [CANDIDATE]: Term 1 election: Requested vote from peers 9ad636a04bcf4998bf21c4126858d557 (127.15.250.132:41747), e520d4b30f204741997deaf911059ce6 (127.15.250.133:45281), eec3c50d1e4043c28a9ec95776302146 (127.15.250.131:35751), cf70a94b41d24bcab02a0e2beebac61d (127.15.250.129:45771)
I20260808 07:03:11.172878 23780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e8975b6bb314806a9b5afdc9a22253a" candidate_uuid: "9e33cc47976341d8898174a800a08ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9ad636a04bcf4998bf21c4126858d557"
I20260808 07:03:11.173023 23780 raft_consensus.cc:3037] T 3e8975b6bb314806a9b5afdc9a22253a P 9ad636a04bcf4998bf21c4126858d557 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:11.173422 23375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e8975b6bb314806a9b5afdc9a22253a" candidate_uuid: "9e33cc47976341d8898174a800a08ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cf70a94b41d24bcab02a0e2beebac61d"
I20260808 07:03:11.173540 23375 raft_consensus.cc:3037] T 3e8975b6bb314806a9b5afdc9a22253a P cf70a94b41d24bcab02a0e2beebac61d [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:11.174031 23780 raft_consensus.cc:2454] T 3e8975b6bb314806a9b5afdc9a22253a P 9ad636a04bcf4998bf21c4126858d557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e33cc47976341d8898174a800a08ff5 in term 1.
I20260808 07:03:11.174506 23375 raft_consensus.cc:2454] T 3e8975b6bb314806a9b5afdc9a22253a P cf70a94b41d24bcab02a0e2beebac61d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e33cc47976341d8898174a800a08ff5 in term 1.
I20260808 07:03:11.174548 23646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e8975b6bb314806a9b5afdc9a22253a" candidate_uuid: "9e33cc47976341d8898174a800a08ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "eec3c50d1e4043c28a9ec95776302146"
I20260808 07:03:11.174650 23918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e8975b6bb314806a9b5afdc9a22253a" candidate_uuid: "9e33cc47976341d8898174a800a08ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e520d4b30f204741997deaf911059ce6"
I20260808 07:03:11.174697 23646 raft_consensus.cc:3037] T 3e8975b6bb314806a9b5afdc9a22253a P eec3c50d1e4043c28a9ec95776302146 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:11.174757 23918 raft_consensus.cc:3037] T 3e8975b6bb314806a9b5afdc9a22253a P e520d4b30f204741997deaf911059ce6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:11.174791 23441 leader_election.cc:302] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 9ad636a04bcf4998bf21c4126858d557, 9e33cc47976341d8898174a800a08ff5, cf70a94b41d24bcab02a0e2beebac61d; no voters:
I20260808 07:03:11.175565 23918 raft_consensus.cc:2454] T 3e8975b6bb314806a9b5afdc9a22253a P e520d4b30f204741997deaf911059ce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e33cc47976341d8898174a800a08ff5 in term 1.
I20260808 07:03:11.175565 23646 raft_consensus.cc:2454] T 3e8975b6bb314806a9b5afdc9a22253a P eec3c50d1e4043c28a9ec95776302146 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e33cc47976341d8898174a800a08ff5 in term 1.
I20260808 07:03:11.175747 24000 raft_consensus.cc:2781] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:11.175916 24000 raft_consensus.cc:686] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5 [term 1 LEADER]: Becoming Leader. State: Replica: 9e33cc47976341d8898174a800a08ff5, State: Running, Role: LEADER
I20260808 07:03:11.176038 24000 consensus_queue.cc:237] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } } peers { permanent_uuid: "eec3c50d1e4043c28a9ec95776302146" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 35751 } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } }
I20260808 07:03:11.177182 23227 catalog_manager.cc:5495] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e33cc47976341d8898174a800a08ff5 (127.15.250.130). New cstate: current_term: 1 leader_uuid: "9e33cc47976341d8898174a800a08ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eec3c50d1e4043c28a9ec95776302146" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 35751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } health_report { overall_health: UNKNOWN } } }
W20260808 07:03:11.190363 23558 tablet.cc:2369] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 07:03:11.203449 23830 tablet.cc:2369] T 3e8975b6bb314806a9b5afdc9a22253a P 9ad636a04bcf4998bf21c4126858d557: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 07:03:11.311440 23694 tablet.cc:2369] T 3e8975b6bb314806a9b5afdc9a22253a P eec3c50d1e4043c28a9ec95776302146: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 07:03:11.313407 23423 tablet.cc:2369] T 3e8975b6bb314806a9b5afdc9a22253a P cf70a94b41d24bcab02a0e2beebac61d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 07:03:11.332815 23968 tablet.cc:2369] T 3e8975b6bb314806a9b5afdc9a22253a P e520d4b30f204741997deaf911059ce6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 07:03:11.553472 24000 consensus_queue.cc:1035] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:03:11.657883 24000 consensus_queue.cc:1035] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eec3c50d1e4043c28a9ec95776302146" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 35751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:03:11.660246 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 23194
I20260808 07:03:11.660737 24017 consensus_queue.cc:1035] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:03:11.661697 24010 consensus_queue.cc:1035] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 07:03:11.667254 23712 connection.cc:535] client connection to 127.15.250.190:34879 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 07:03:11.667246 23576 connection.cc:535] client connection to 127.15.250.190:34879 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 07:03:11.667225 23849 connection.cc:535] client connection to 127.15.250.190:34879 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 07:03:11.667444 23693 heartbeater.cc:644] Failed to heartbeat to 127.15.250.190:34879 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 07:03:11.667454 23829 heartbeater.cc:644] Failed to heartbeat to 127.15.250.190:34879 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 07:03:11.667460 23967 heartbeater.cc:644] Failed to heartbeat to 127.15.250.190:34879 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 07:03:11.667719 23557 heartbeater.cc:644] Failed to heartbeat to 127.15.250.190:34879 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.15.250.190:34879 (error 108)
W20260808 07:03:11.667794 23306 connection.cc:535] client connection to 127.15.250.190:34879 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 07:03:11.667966 23422 heartbeater.cc:644] Failed to heartbeat to 127.15.250.190:34879 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 07:03:11.668308 24008 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 07:03:11.667946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.250.190:34879 (local address 127.15.250.130:37811)
0808 07:03:11.668004 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0808 07:03:11.668031 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 07:03:11.668077 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.250.190:34879: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":38}
I20260808 07:03:11.668408 16362 raft_consensus_election-itest.cc:666] Removing tserver with uuid eec3c50d1e4043c28a9ec95776302146
W20260808 07:03:11.668623 24027 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 07:03:11.667817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.250.190:34879 (local address 127.15.250.133:34069)
0808 07:03:11.668434 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0808 07:03:11.668477 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.250.190:34879: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":66,"threads_started":1}
I20260808 07:03:11.669031 23510 consensus_queue.cc:237] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } }
I20260808 07:03:11.669637 23918 raft_consensus.cc:1261] T 3e8975b6bb314806a9b5afdc9a22253a P e520d4b30f204741997deaf911059ce6 [term 1 FOLLOWER]: Refusing update from remote peer 9e33cc47976341d8898174a800a08ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 07:03:11.669697 23780 raft_consensus.cc:1261] T 3e8975b6bb314806a9b5afdc9a22253a P 9ad636a04bcf4998bf21c4126858d557 [term 1 FOLLOWER]: Refusing update from remote peer 9e33cc47976341d8898174a800a08ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 07:03:11.669739 23375 raft_consensus.cc:1261] T 3e8975b6bb314806a9b5afdc9a22253a P cf70a94b41d24bcab02a0e2beebac61d [term 1 FOLLOWER]: Refusing update from remote peer 9e33cc47976341d8898174a800a08ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 07:03:11.669875 24017 consensus_queue.cc:1035] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 07:03:11.669947 24010 consensus_queue.cc:1035] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 07:03:11.670246 24018 consensus_queue.cc:1035] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 07:03:11.671010 24018 raft_consensus.cc:2932] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER eec3c50d1e4043c28a9ec95776302146 (127.15.250.131) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } } }
I20260808 07:03:11.671317 23918 raft_consensus.cc:2932] T 3e8975b6bb314806a9b5afdc9a22253a P e520d4b30f204741997deaf911059ce6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER eec3c50d1e4043c28a9ec95776302146 (127.15.250.131) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } } }
I20260808 07:03:11.672372 23375 raft_consensus.cc:2932] T 3e8975b6bb314806a9b5afdc9a22253a P cf70a94b41d24bcab02a0e2beebac61d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER eec3c50d1e4043c28a9ec95776302146 (127.15.250.131) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } } }
I20260808 07:03:11.672418 23780 raft_consensus.cc:2932] T 3e8975b6bb314806a9b5afdc9a22253a P 9ad636a04bcf4998bf21c4126858d557 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER eec3c50d1e4043c28a9ec95776302146 (127.15.250.131) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } } }
I20260808 07:03:11.683568 16362 raft_consensus_election-itest.cc:676] Pausing 3 replicas...
I20260808 07:03:11.715335 16362 raft_consensus_election-itest.cc:684] Adding back Peer eec3c50d1e4043c28a9ec95776302146 and expecting timeout...
I20260808 07:03:11.715981 23510 consensus_queue.cc:237] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } } peers { permanent_uuid: "eec3c50d1e4043c28a9ec95776302146" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 35751 } attrs { } }
I20260808 07:03:11.716799 23646 raft_consensus.cc:1261] T 3e8975b6bb314806a9b5afdc9a22253a P eec3c50d1e4043c28a9ec95776302146 [term 1 FOLLOWER]: Refusing update from remote peer 9e33cc47976341d8898174a800a08ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 07:03:11.717047 24010 consensus_queue.cc:1035] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eec3c50d1e4043c28a9ec95776302146" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 35751 } attrs { }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 07:03:11.717803 23646 raft_consensus.cc:858] T 3e8975b6bb314806a9b5afdc9a22253a P eec3c50d1e4043c28a9ec95776302146 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } } peers { permanent_uuid: "eec3c50d1e4043c28a9ec95776302146" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 35751 } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } }.
Pending config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } }:
New config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } } peers { permanent_uuid: "eec3c50d1e4043c28a9ec95776302146" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 35751 } attrs { } }
W20260808 07:03:11.717864 23646 raft_consensus.cc:1565] T 3e8975b6bb314806a9b5afdc9a22253a P eec3c50d1e4043c28a9ec95776302146 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 3' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } } peers { permanent_uuid: "eec3c50d1e4043c28a9ec95776302146" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 35751 } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } }.
Pending config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } }:
New config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } } peers { permanent_uuid: "eec3c50d1e4043c28a9ec95776302146" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 35751 } attrs { } }
I20260808 07:03:11.717967 23646 raft_consensus.cc:2932] T 3e8975b6bb314806a9b5afdc9a22253a P eec3c50d1e4043c28a9ec95776302146 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER eec3c50d1e4043c28a9ec95776302146 (127.15.250.131) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } } }
I20260808 07:03:11.719223 23646 raft_consensus.cc:1467] T 3e8975b6bb314806a9b5afdc9a22253a P eec3c50d1e4043c28a9ec95776302146 [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20260808 07:03:11.816350 16362 raft_consensus_election-itest.cc:688] Timeout achieved.
I20260808 07:03:11.820538 16362 raft_consensus_election-itest.cc:699] Forcing Peer 9e33cc47976341d8898174a800a08ff5 to step down...
I20260808 07:03:11.820885 23510 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3e8975b6bb314806a9b5afdc9a22253a"
dest_uuid: "9e33cc47976341d8898174a800a08ff5"
from {username='slave'} at 127.0.0.1:34020
I20260808 07:03:11.820986 23510 raft_consensus.cc:580] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5 [term 1 LEADER]: Received request to step down
I20260808 07:03:11.821005 23510 raft_consensus.cc:3032] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 07:03:11.821017 23510 raft_consensus.cc:729] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9e33cc47976341d8898174a800a08ff5, State: Running, Role: LEADER
I20260808 07:03:11.821055 23510 consensus_queue.cc:260] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } } peers { permanent_uuid: "eec3c50d1e4043c28a9ec95776302146" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 35751 } attrs { } }
I20260808 07:03:11.821133 23510 raft_consensus.cc:3037] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 07:03:11.821385 16362 raft_consensus_election-itest.cc:705] Resuming Peer cf70a94b41d24bcab02a0e2beebac61d ...
I20260808 07:03:11.821946 16362 raft_consensus_election-itest.cc:710] Trying to elect Peer eec3c50d1e4043c28a9ec95776302146 ...
I20260808 07:03:11.822505 23375 raft_consensus.cc:1261] T 3e8975b6bb314806a9b5afdc9a22253a P cf70a94b41d24bcab02a0e2beebac61d [term 1 FOLLOWER]: Refusing update from remote peer 9e33cc47976341d8898174a800a08ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 07:03:11.822497 23646 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3e8975b6bb314806a9b5afdc9a22253a"
dest_uuid: "eec3c50d1e4043c28a9ec95776302146"
from {username='slave'} at 127.0.0.1:53194
I20260808 07:03:11.822616 23646 raft_consensus.cc:482] T 3e8975b6bb314806a9b5afdc9a22253a P eec3c50d1e4043c28a9ec95776302146 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 07:03:11.822659 23646 raft_consensus.cc:3037] T 3e8975b6bb314806a9b5afdc9a22253a P eec3c50d1e4043c28a9ec95776302146 [term 1 FOLLOWER]: Advancing to term 2
I20260808 07:03:11.823292 23646 raft_consensus.cc:504] T 3e8975b6bb314806a9b5afdc9a22253a P eec3c50d1e4043c28a9ec95776302146 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } } peers { permanent_uuid: "eec3c50d1e4043c28a9ec95776302146" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 35751 } attrs { } }
I20260808 07:03:11.823643 23646 leader_election.cc:288] T 3e8975b6bb314806a9b5afdc9a22253a P eec3c50d1e4043c28a9ec95776302146 [CANDIDATE]: Term 2 election: Requested vote from peers 9ad636a04bcf4998bf21c4126858d557 (127.15.250.132:41747), 9e33cc47976341d8898174a800a08ff5 (127.15.250.130:41633), e520d4b30f204741997deaf911059ce6 (127.15.250.133:45281), cf70a94b41d24bcab02a0e2beebac61d (127.15.250.129:45771)
I20260808 07:03:11.825497 16362 cluster_itest_util.cc:256] Not converged past 4 yet: 1.2 1.3 1.3
I20260808 07:03:11.827430 23375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e8975b6bb314806a9b5afdc9a22253a" candidate_uuid: "eec3c50d1e4043c28a9ec95776302146" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf70a94b41d24bcab02a0e2beebac61d"
I20260808 07:03:11.827507 23375 raft_consensus.cc:1799] T 3e8975b6bb314806a9b5afdc9a22253a P cf70a94b41d24bcab02a0e2beebac61d [term 1 FOLLOWER]: Handling vote request from an unknown peer eec3c50d1e4043c28a9ec95776302146
I20260808 07:03:11.827526 23375 raft_consensus.cc:3037] T 3e8975b6bb314806a9b5afdc9a22253a P cf70a94b41d24bcab02a0e2beebac61d [term 1 FOLLOWER]: Advancing to term 2
I20260808 07:03:11.827453 23510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e8975b6bb314806a9b5afdc9a22253a" candidate_uuid: "eec3c50d1e4043c28a9ec95776302146" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9e33cc47976341d8898174a800a08ff5"
I20260808 07:03:11.828120 23375 raft_consensus.cc:2454] T 3e8975b6bb314806a9b5afdc9a22253a P cf70a94b41d24bcab02a0e2beebac61d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eec3c50d1e4043c28a9ec95776302146 in term 2.
I20260808 07:03:11.828126 23510 raft_consensus.cc:2454] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eec3c50d1e4043c28a9ec95776302146 in term 2.
I20260808 07:03:11.828353 23578 leader_election.cc:302] T 3e8975b6bb314806a9b5afdc9a22253a P eec3c50d1e4043c28a9ec95776302146 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 9e33cc47976341d8898174a800a08ff5, cf70a94b41d24bcab02a0e2beebac61d, eec3c50d1e4043c28a9ec95776302146; no voters:
I20260808 07:03:11.828488 24013 raft_consensus.cc:2781] T 3e8975b6bb314806a9b5afdc9a22253a P eec3c50d1e4043c28a9ec95776302146 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 07:03:11.828639 24013 raft_consensus.cc:686] T 3e8975b6bb314806a9b5afdc9a22253a P eec3c50d1e4043c28a9ec95776302146 [term 2 LEADER]: Becoming Leader. State: Replica: eec3c50d1e4043c28a9ec95776302146, State: Running, Role: LEADER
I20260808 07:03:11.828748 24013 consensus_queue.cc:237] T 3e8975b6bb314806a9b5afdc9a22253a P eec3c50d1e4043c28a9ec95776302146 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } } peers { permanent_uuid: "eec3c50d1e4043c28a9ec95776302146" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 35751 } attrs { } }
I20260808 07:03:11.927538 16362 cluster_itest_util.cc:256] Not converged past 4 yet: 1.2 1.3 2.4
I20260808 07:03:12.129310 16362 cluster_itest_util.cc:256] Not converged past 4 yet: 1.2 1.3 2.4
I20260808 07:03:12.269985 23375 raft_consensus.cc:1261] T 3e8975b6bb314806a9b5afdc9a22253a P cf70a94b41d24bcab02a0e2beebac61d [term 2 FOLLOWER]: Refusing update from remote peer eec3c50d1e4043c28a9ec95776302146: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 07:03:12.270424 24013 consensus_queue.cc:1035] T 3e8975b6bb314806a9b5afdc9a22253a P eec3c50d1e4043c28a9ec95776302146 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 07:03:12.358836 23510 raft_consensus.cc:1261] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5 [term 2 FOLLOWER]: Refusing update from remote peer eec3c50d1e4043c28a9ec95776302146: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 07:03:12.359274 24013 consensus_queue.cc:1035] T 3e8975b6bb314806a9b5afdc9a22253a P eec3c50d1e4043c28a9ec95776302146 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 07:03:12.360651 24045 raft_consensus.cc:2932] T 3e8975b6bb314806a9b5afdc9a22253a P eec3c50d1e4043c28a9ec95776302146 [term 2 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER eec3c50d1e4043c28a9ec95776302146 (127.15.250.131) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } } peers { permanent_uuid: "eec3c50d1e4043c28a9ec95776302146" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 35751 } attrs { } } }
I20260808 07:03:12.360833 23510 raft_consensus.cc:2932] T 3e8975b6bb314806a9b5afdc9a22253a P 9e33cc47976341d8898174a800a08ff5 [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER eec3c50d1e4043c28a9ec95776302146 (127.15.250.131) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } } peers { permanent_uuid: "eec3c50d1e4043c28a9ec95776302146" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 35751 } attrs { } } }
W20260808 07:03:12.361631 23510 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:34020 (request call id 50) took 645 ms (client timeout 100 ms). Trace:
W20260808 07:03:12.361708 23510 rpcz_store.cc:269] 0808 07:03:11.715754 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:03:11.715811 (+ 57us) service_pool.cc:227] Handling call
0808 07:03:11.716366 (+ 555us) log.cc:842] Serialized 618 byte log entry
0808 07:03:12.361622 (+645256us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1,"raft.queue_time_us":244,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":132,"spinlock_wait_cycles":11264}
I20260808 07:03:12.362356 23375 raft_consensus.cc:2932] T 3e8975b6bb314806a9b5afdc9a22253a P cf70a94b41d24bcab02a0e2beebac61d [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER eec3c50d1e4043c28a9ec95776302146 (127.15.250.131) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } } peers { permanent_uuid: "eec3c50d1e4043c28a9ec95776302146" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 35751 } attrs { } } }
I20260808 07:03:12.431001 16362 raft_consensus_election-itest.cc:716] Resuming remaining nodes...
I20260808 07:03:12.431810 23780 raft_consensus.cc:1261] T 3e8975b6bb314806a9b5afdc9a22253a P 9ad636a04bcf4998bf21c4126858d557 [term 1 FOLLOWER]: Refusing update from remote peer 9e33cc47976341d8898174a800a08ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 07:03:12.433374 23918 raft_consensus.cc:1261] T 3e8975b6bb314806a9b5afdc9a22253a P e520d4b30f204741997deaf911059ce6 [term 1 FOLLOWER]: Refusing update from remote peer 9e33cc47976341d8898174a800a08ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 07:03:12.438441 23918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e8975b6bb314806a9b5afdc9a22253a" candidate_uuid: "eec3c50d1e4043c28a9ec95776302146" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e520d4b30f204741997deaf911059ce6"
I20260808 07:03:12.438552 23918 raft_consensus.cc:1799] T 3e8975b6bb314806a9b5afdc9a22253a P e520d4b30f204741997deaf911059ce6 [term 1 FOLLOWER]: Handling vote request from an unknown peer eec3c50d1e4043c28a9ec95776302146
I20260808 07:03:12.438584 23918 raft_consensus.cc:3037] T 3e8975b6bb314806a9b5afdc9a22253a P e520d4b30f204741997deaf911059ce6 [term 1 FOLLOWER]: Advancing to term 2
I20260808 07:03:12.439483 23918 raft_consensus.cc:2454] T 3e8975b6bb314806a9b5afdc9a22253a P e520d4b30f204741997deaf911059ce6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eec3c50d1e4043c28a9ec95776302146 in term 2.
I20260808 07:03:12.439632 23917 raft_consensus.cc:1261] T 3e8975b6bb314806a9b5afdc9a22253a P e520d4b30f204741997deaf911059ce6 [term 2 FOLLOWER]: Refusing update from remote peer eec3c50d1e4043c28a9ec95776302146: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 07:03:12.439841 24047 consensus_queue.cc:1035] T 3e8975b6bb314806a9b5afdc9a22253a P eec3c50d1e4043c28a9ec95776302146 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 07:03:12.440369 16362 cluster_itest_util.cc:256] Not converged past 5 yet: 2.5 2.5 1.2 1.2 2.5
I20260808 07:03:12.440986 23779 raft_consensus.cc:3037] T 3e8975b6bb314806a9b5afdc9a22253a P 9ad636a04bcf4998bf21c4126858d557 [term 1 FOLLOWER]: Advancing to term 2
I20260808 07:03:12.441071 23917 raft_consensus.cc:2932] T 3e8975b6bb314806a9b5afdc9a22253a P e520d4b30f204741997deaf911059ce6 [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER eec3c50d1e4043c28a9ec95776302146 (127.15.250.131) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } } peers { permanent_uuid: "eec3c50d1e4043c28a9ec95776302146" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 35751 } attrs { } } }
I20260808 07:03:12.441660 23779 raft_consensus.cc:1261] T 3e8975b6bb314806a9b5afdc9a22253a P 9ad636a04bcf4998bf21c4126858d557 [term 2 FOLLOWER]: Refusing update from remote peer eec3c50d1e4043c28a9ec95776302146: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 07:03:12.441880 24047 consensus_queue.cc:1035] T 3e8975b6bb314806a9b5afdc9a22253a P eec3c50d1e4043c28a9ec95776302146 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 07:03:12.442240 23780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e8975b6bb314806a9b5afdc9a22253a" candidate_uuid: "eec3c50d1e4043c28a9ec95776302146" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9ad636a04bcf4998bf21c4126858d557"
I20260808 07:03:12.442845 23779 raft_consensus.cc:2932] T 3e8975b6bb314806a9b5afdc9a22253a P 9ad636a04bcf4998bf21c4126858d557 [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER eec3c50d1e4043c28a9ec95776302146 (127.15.250.131) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ad636a04bcf4998bf21c4126858d557" member_type: VOTER last_known_addr { host: "127.15.250.132" port: 41747 } } peers { permanent_uuid: "9e33cc47976341d8898174a800a08ff5" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 41633 } } peers { permanent_uuid: "e520d4b30f204741997deaf911059ce6" member_type: VOTER last_known_addr { host: "127.15.250.133" port: 45281 } } peers { permanent_uuid: "cf70a94b41d24bcab02a0e2beebac61d" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45771 } } peers { permanent_uuid: "eec3c50d1e4043c28a9ec95776302146" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 35751 } attrs { } } }
I20260808 07:03:12.443914 23780 raft_consensus.cc:2396] T 3e8975b6bb314806a9b5afdc9a22253a P 9ad636a04bcf4998bf21c4126858d557 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate eec3c50d1e4043c28a9ec95776302146 for term 2 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260808 07:03:12.542958 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 23291
I20260808 07:03:12.549499 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 23425
I20260808 07:03:12.555411 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 23560
I20260808 07:03:12.560648 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 23696
I20260808 07:03:12.566009 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 23832
2026-08-08T07:03:12Z chronyd exiting
[ OK ] RaftConsensusElectionITest.ElectPendingVoter (2322 ms)
[ RUN ] RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy
I20260808 07:03:12.587423 16362 test_util.cc:268] Using random seed: -1226603877
I20260808 07:03:12.587980 16362 ts_itest-base.cc:112] Starting cluster with:
I20260808 07:03:12.588035 16362 ts_itest-base.cc:113] --------------
I20260808 07:03:12.588057 16362 ts_itest-base.cc:114] 3 tablet servers
I20260808 07:03:12.588084 16362 ts_itest-base.cc:115] 3 replicas per TS
I20260808 07:03:12.588109 16362 ts_itest-base.cc:116] --------------
2026-08-08T07:03:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:03:12Z Disabled control of system clock
I20260808 07:03:12.597469 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.250.190:35135
--webserver_interface=127.15.250.190
--webserver_port=0
--builtin_ntp_servers=127.15.250.148:41221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.250.190:35135
--master_add_server_when_underreplicated=false
--raft_prepare_replacement_before_eviction=false with env {}
W20260808 07:03:12.679620 24075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:12.679872 24075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:12.679908 24075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:12.681105 24075 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260808 07:03:12.681185 24075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:03:12.681210 24075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:12.681242 24075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:03:12.681267 24075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:03:12.682672 24075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:41221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
--master_add_server_when_underreplicated=false
--ipki_ca_key_size=768
--master_addresses=127.15.250.190:35135
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.250.190:35135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.250.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.24075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:12.682963 24075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:12.683275 24075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:12.688481 24080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:12.688482 24082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:12.688679 24075 server_base.cc:1034] running on GCE node
W20260808 07:03:12.688517 24084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:12.689036 24075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:12.689296 24075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:12.690486 24075 hybrid_clock.cc:648] HybridClock initialized: now 1786172592690470 us; error 44 us; skew 500 ppm
I20260808 07:03:12.696695 24075 webserver.cc:460] Webserver started at http://127.15.250.190:36173/ using document root <none> and password file <none>
I20260808 07:03:12.696995 24075 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:12.697072 24075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:12.697227 24075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:12.698258 24075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "74682779344e47ecbb8b8e06a774afc5"
format_stamp: "Formatted at 2026-08-08 07:03:12 on dist-test-slave-0b58"
I20260808 07:03:12.698642 24075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "74682779344e47ecbb8b8e06a774afc5"
format_stamp: "Formatted at 2026-08-08 07:03:12 on dist-test-slave-0b58"
I20260808 07:03:12.699994 24075 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 07:03:12.700843 24091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:12.701035 24075 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 07:03:12.701131 24075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
uuid: "74682779344e47ecbb8b8e06a774afc5"
format_stamp: "Formatted at 2026-08-08 07:03:12 on dist-test-slave-0b58"
I20260808 07:03:12.701216 24075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:12.719955 24075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:12.720352 24075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:12.720515 24075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:12.724802 24075 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.190:35135
I20260808 07:03:12.724861 24143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.190:35135 every 8 connection(s)
I20260808 07:03:12.725205 24075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260808 07:03:12.725888 24144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:12.728804 24144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5: Bootstrap starting.
I20260808 07:03:12.729436 24144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:12.729810 24144 log.cc:824] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:12.730553 24144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5: No bootstrap required, opened a new log
I20260808 07:03:12.732399 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 24075
I20260808 07:03:12.732306 24144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74682779344e47ecbb8b8e06a774afc5" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 35135 } }
I20260808 07:03:12.732506 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260808 07:03:12.732496 24144 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:03:12.732535 24144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74682779344e47ecbb8b8e06a774afc5, State: Initialized, Role: FOLLOWER
I20260808 07:03:12.732717 24144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74682779344e47ecbb8b8e06a774afc5" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 35135 } }
I20260808 07:03:12.732824 24144 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:03:12.732860 24144 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:03:12.732918 24144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:12.733698 24144 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74682779344e47ecbb8b8e06a774afc5" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 35135 } }
I20260808 07:03:12.733811 24144 leader_election.cc:302] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74682779344e47ecbb8b8e06a774afc5; no voters:
I20260808 07:03:12.733968 24144 leader_election.cc:288] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:03:12.734023 24150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:12.734217 24150 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [term 1 LEADER]: Becoming Leader. State: Replica: 74682779344e47ecbb8b8e06a774afc5, State: Running, Role: LEADER
I20260808 07:03:12.734318 24150 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74682779344e47ecbb8b8e06a774afc5" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 35135 } }
I20260808 07:03:12.734361 24144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:03:12.734668 24150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74682779344e47ecbb8b8e06a774afc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74682779344e47ecbb8b8e06a774afc5" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 35135 } } }
I20260808 07:03:12.734827 24150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [sys.catalog]: This master's current role is: LEADER
I20260808 07:03:12.735013 24150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74682779344e47ecbb8b8e06a774afc5. Latest consensus state: current_term: 1 leader_uuid: "74682779344e47ecbb8b8e06a774afc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74682779344e47ecbb8b8e06a774afc5" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 35135 } } }
I20260808 07:03:12.735126 24150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [sys.catalog]: This master's current role is: LEADER
I20260808 07:03:12.735437 24157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:03:12.736068 24157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:03:12.738040 24157 catalog_manager.cc:1347] Generated new cluster ID: 9d7ddfb7f09f44cca0fa054921cee053
I20260808 07:03:12.738104 24157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:03:12.746794 24157 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:03:12.747519 24157 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:03:12.759873 24157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5: Generated new TSK 0
I20260808 07:03:12.760104 24157 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:03:12.764871 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.129:0
--local_ip_for_outbound_sockets=127.15.250.129
--webserver_interface=127.15.250.129
--webserver_port=0
--tserver_master_addrs=127.15.250.190:35135
--builtin_ntp_servers=127.15.250.148:41221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
W20260808 07:03:12.858105 24169 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20260808 07:03:12.858520 24169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:12.858582 24169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:12.858623 24169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:12.859954 24169 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20260808 07:03:12.860059 24169 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20260808 07:03:12.860088 24169 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260808 07:03:12.860134 24169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:12.860181 24169 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20260808 07:03:12.860229 24169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.129
I20260808 07:03:12.861694 24169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:41221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.250.129
--webserver_port=0
--tserver_master_addrs=127.15.250.190:35135
--never_fsync=true
--heap_profile_path=/tmp/kudu.24169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.129
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:12.861984 24169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:12.862311 24169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:12.868217 24175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:12.868219 24174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:12.868417 24169 server_base.cc:1034] running on GCE node
W20260808 07:03:12.868244 24177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:12.868724 24169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:12.869004 24169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:12.870213 24169 hybrid_clock.cc:648] HybridClock initialized: now 1786172592870188 us; error 41 us; skew 500 ppm
I20260808 07:03:12.871583 24169 webserver.cc:460] Webserver started at http://127.15.250.129:35735/ using document root <none> and password file <none>
I20260808 07:03:12.871841 24169 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:12.871914 24169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:12.872074 24169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:12.873149 24169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "dc14e3d81a4e4d44bf00163aaab40610"
format_stamp: "Formatted at 2026-08-08 07:03:12 on dist-test-slave-0b58"
I20260808 07:03:12.873533 24169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "dc14e3d81a4e4d44bf00163aaab40610"
format_stamp: "Formatted at 2026-08-08 07:03:12 on dist-test-slave-0b58"
I20260808 07:03:12.874900 24169 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 07:03:12.875687 24183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:12.875883 24169 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 07:03:12.875974 24169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "dc14e3d81a4e4d44bf00163aaab40610"
format_stamp: "Formatted at 2026-08-08 07:03:12 on dist-test-slave-0b58"
I20260808 07:03:12.876055 24169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:12.892830 24169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:12.893172 24169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:12.893333 24169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:12.893579 24169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:03:12.893942 24169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:03:12.893986 24169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:12.894039 24169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:03:12.894064 24169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:12.900209 24169 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.129:43635
I20260808 07:03:12.900295 24298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.129:43635 every 8 connection(s)
I20260808 07:03:12.900719 24169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260808 07:03:12.901562 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 24169
I20260808 07:03:12.901669 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260808 07:03:12.903095 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.130:0
--local_ip_for_outbound_sockets=127.15.250.130
--webserver_interface=127.15.250.130
--webserver_port=0
--tserver_master_addrs=127.15.250.190:35135
--builtin_ntp_servers=127.15.250.148:41221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20260808 07:03:12.907920 24299 heartbeater.cc:347] Connected to a master server at 127.15.250.190:35135
I20260808 07:03:12.908012 24299 heartbeater.cc:464] Registering TS with master...
I20260808 07:03:12.908397 24299 heartbeater.cc:511] Master 127.15.250.190:35135 requested a full tablet report, sending...
I20260808 07:03:12.908972 24108 ts_manager.cc:194] Registered new tserver with Master: dc14e3d81a4e4d44bf00163aaab40610 (127.15.250.129:43635)
I20260808 07:03:12.909626 24108 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.129:51391
W20260808 07:03:12.986738 24302 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20260808 07:03:12.987115 24302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:12.987154 24302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:12.987188 24302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:12.988595 24302 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20260808 07:03:12.988654 24302 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20260808 07:03:12.988682 24302 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260808 07:03:12.988718 24302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:12.988759 24302 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20260808 07:03:12.988790 24302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.130
I20260808 07:03:12.990142 24302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:41221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.250.130
--webserver_port=0
--tserver_master_addrs=127.15.250.190:35135
--never_fsync=true
--heap_profile_path=/tmp/kudu.24302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.130
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:12.990410 24302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:12.990708 24302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:12.995820 24310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:12.995816 24309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:12.995820 24312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:12.996080 24302 server_base.cc:1034] running on GCE node
I20260808 07:03:12.996407 24302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:12.996671 24302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:12.997828 24302 hybrid_clock.cc:648] HybridClock initialized: now 1786172592997806 us; error 39 us; skew 500 ppm
I20260808 07:03:12.999264 24302 webserver.cc:460] Webserver started at http://127.15.250.130:44221/ using document root <none> and password file <none>
I20260808 07:03:12.999519 24302 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:12.999567 24302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:12.999752 24302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:13.000873 24302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "0383e7089e6448eb84566b41e2415e2e"
format_stamp: "Formatted at 2026-08-08 07:03:12 on dist-test-slave-0b58"
I20260808 07:03:13.001302 24302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "0383e7089e6448eb84566b41e2415e2e"
format_stamp: "Formatted at 2026-08-08 07:03:12 on dist-test-slave-0b58"
I20260808 07:03:13.002679 24302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 07:03:13.003582 24318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:13.003772 24302 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:03:13.003883 24302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "0383e7089e6448eb84566b41e2415e2e"
format_stamp: "Formatted at 2026-08-08 07:03:12 on dist-test-slave-0b58"
I20260808 07:03:13.003964 24302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:13.022619 24302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:13.022965 24302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:13.023128 24302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:13.023391 24302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:03:13.023743 24302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:03:13.023802 24302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:13.023844 24302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:03:13.023872 24302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:13.029917 24302 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.130:34401
I20260808 07:03:13.029963 24436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.130:34401 every 8 connection(s)
I20260808 07:03:13.030328 24302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260808 07:03:13.036142 24437 heartbeater.cc:347] Connected to a master server at 127.15.250.190:35135
I20260808 07:03:13.036240 24437 heartbeater.cc:464] Registering TS with master...
I20260808 07:03:13.036439 24437 heartbeater.cc:511] Master 127.15.250.190:35135 requested a full tablet report, sending...
I20260808 07:03:13.036821 24108 ts_manager.cc:194] Registered new tserver with Master: 0383e7089e6448eb84566b41e2415e2e (127.15.250.130:34401)
I20260808 07:03:13.037204 24108 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.130:42869
I20260808 07:03:13.039826 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 24302
I20260808 07:03:13.039919 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260808 07:03:13.040997 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.131:0
--local_ip_for_outbound_sockets=127.15.250.131
--webserver_interface=127.15.250.131
--webserver_port=0
--tserver_master_addrs=127.15.250.190:35135
--builtin_ntp_servers=127.15.250.148:41221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
W20260808 07:03:13.124079 24440 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20260808 07:03:13.124387 24440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:13.124418 24440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:13.124456 24440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:13.125687 24440 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20260808 07:03:13.125759 24440 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20260808 07:03:13.125785 24440 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260808 07:03:13.125810 24440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:13.125855 24440 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20260808 07:03:13.125880 24440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.131
I20260808 07:03:13.127297 24440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:41221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.250.131
--webserver_port=0
--tserver_master_addrs=127.15.250.190:35135
--never_fsync=true
--heap_profile_path=/tmp/kudu.24440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.131
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:13.127581 24440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:13.127956 24440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:13.133059 24446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:13.133112 24445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:13.133070 24448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:13.133319 24440 server_base.cc:1034] running on GCE node
I20260808 07:03:13.133610 24440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:13.133919 24440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:13.135058 24440 hybrid_clock.cc:648] HybridClock initialized: now 1786172593135036 us; error 40 us; skew 500 ppm
I20260808 07:03:13.136482 24440 webserver.cc:460] Webserver started at http://127.15.250.131:41489/ using document root <none> and password file <none>
I20260808 07:03:13.136726 24440 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:13.136775 24440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:13.136953 24440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:13.137965 24440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "c24e263e9ede48a0bc8c1e84fb8245e7"
format_stamp: "Formatted at 2026-08-08 07:03:13 on dist-test-slave-0b58"
I20260808 07:03:13.138365 24440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "c24e263e9ede48a0bc8c1e84fb8245e7"
format_stamp: "Formatted at 2026-08-08 07:03:13 on dist-test-slave-0b58"
I20260808 07:03:13.139662 24440 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 07:03:13.140411 24454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:13.140588 24440 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:03:13.140694 24440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "c24e263e9ede48a0bc8c1e84fb8245e7"
format_stamp: "Formatted at 2026-08-08 07:03:13 on dist-test-slave-0b58"
I20260808 07:03:13.140777 24440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:13.166512 24440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:13.166878 24440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:13.167043 24440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:13.167309 24440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:03:13.167691 24440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:03:13.167748 24440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:13.167784 24440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:03:13.167829 24440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:13.174157 24440 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.131:39239
I20260808 07:03:13.174206 24570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.131:39239 every 8 connection(s)
I20260808 07:03:13.174649 24440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260808 07:03:13.176926 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 24440
I20260808 07:03:13.177023 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260808 07:03:13.180931 24571 heartbeater.cc:347] Connected to a master server at 127.15.250.190:35135
I20260808 07:03:13.181057 24571 heartbeater.cc:464] Registering TS with master...
I20260808 07:03:13.181226 24571 heartbeater.cc:511] Master 127.15.250.190:35135 requested a full tablet report, sending...
I20260808 07:03:13.181627 24108 ts_manager.cc:194] Registered new tserver with Master: c24e263e9ede48a0bc8c1e84fb8245e7 (127.15.250.131:39239)
I20260808 07:03:13.181980 24108 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.131:59719
I20260808 07:03:13.189381 16362 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 07:03:13.196000 24108 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34642:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20260808 07:03:13.205086 24233 tablet_service.cc:1467] Processing CreateTablet for tablet 20fa711728674c50a616400f2fc53ca1 (DEFAULT_TABLE table=TestTable [id=5209b2a3bff141b99389ae23512a4cdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:03:13.205260 24368 tablet_service.cc:1467] Processing CreateTablet for tablet 20fa711728674c50a616400f2fc53ca1 (DEFAULT_TABLE table=TestTable [id=5209b2a3bff141b99389ae23512a4cdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:03:13.205507 24233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20fa711728674c50a616400f2fc53ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:13.205550 24368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20fa711728674c50a616400f2fc53ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:13.208323 24502 tablet_service.cc:1467] Processing CreateTablet for tablet 20fa711728674c50a616400f2fc53ca1 (DEFAULT_TABLE table=TestTable [id=5209b2a3bff141b99389ae23512a4cdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:03:13.208662 24502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20fa711728674c50a616400f2fc53ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:13.208915 24586 tablet_bootstrap.cc:492] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: Bootstrap starting.
I20260808 07:03:13.209125 24585 tablet_bootstrap.cc:492] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e: Bootstrap starting.
I20260808 07:03:13.209825 24585 tablet_bootstrap.cc:654] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:13.210165 24585 log.cc:824] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:13.210871 24585 tablet_bootstrap.cc:492] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e: No bootstrap required, opened a new log
I20260808 07:03:13.210960 24585 ts_tablet_manager.cc:1397] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:03:13.211076 24586 tablet_bootstrap.cc:654] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:13.211409 24586 log.cc:824] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:13.212139 24589 tablet_bootstrap.cc:492] T 20fa711728674c50a616400f2fc53ca1 P c24e263e9ede48a0bc8c1e84fb8245e7: Bootstrap starting.
I20260808 07:03:13.212395 24586 tablet_bootstrap.cc:492] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: No bootstrap required, opened a new log
I20260808 07:03:13.212483 24586 ts_tablet_manager.cc:1397] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260808 07:03:13.212916 24589 tablet_bootstrap.cc:654] T 20fa711728674c50a616400f2fc53ca1 P c24e263e9ede48a0bc8c1e84fb8245e7: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:13.213234 24589 log.cc:824] T 20fa711728674c50a616400f2fc53ca1 P c24e263e9ede48a0bc8c1e84fb8245e7: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:13.213811 24585 raft_consensus.cc:348] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0383e7089e6448eb84566b41e2415e2e" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34401 } } peers { permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 } } peers { permanent_uuid: "dc14e3d81a4e4d44bf00163aaab40610" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 43635 } }
I20260808 07:03:13.213975 24589 tablet_bootstrap.cc:492] T 20fa711728674c50a616400f2fc53ca1 P c24e263e9ede48a0bc8c1e84fb8245e7: No bootstrap required, opened a new log
I20260808 07:03:13.213985 24585 raft_consensus.cc:374] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:03:13.214023 24585 raft_consensus.cc:729] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0383e7089e6448eb84566b41e2415e2e, State: Initialized, Role: FOLLOWER
I20260808 07:03:13.214035 24589 ts_tablet_manager.cc:1397] T 20fa711728674c50a616400f2fc53ca1 P c24e263e9ede48a0bc8c1e84fb8245e7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:03:13.214174 24585 consensus_queue.cc:260] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0383e7089e6448eb84566b41e2415e2e" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34401 } } peers { permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 } } peers { permanent_uuid: "dc14e3d81a4e4d44bf00163aaab40610" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 43635 } }
I20260808 07:03:13.214469 24585 ts_tablet_manager.cc:1428] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260808 07:03:13.214493 24437 heartbeater.cc:503] Master 127.15.250.190:35135 was elected leader, sending a full tablet report...
I20260808 07:03:13.215183 24586 raft_consensus.cc:348] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0383e7089e6448eb84566b41e2415e2e" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34401 } } peers { permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 } } peers { permanent_uuid: "dc14e3d81a4e4d44bf00163aaab40610" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 43635 } }
I20260808 07:03:13.215374 24586 raft_consensus.cc:374] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:03:13.215426 24586 raft_consensus.cc:729] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc14e3d81a4e4d44bf00163aaab40610, State: Initialized, Role: FOLLOWER
I20260808 07:03:13.215562 24586 consensus_queue.cc:260] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0383e7089e6448eb84566b41e2415e2e" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34401 } } peers { permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 } } peers { permanent_uuid: "dc14e3d81a4e4d44bf00163aaab40610" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 43635 } }
I20260808 07:03:13.215845 24586 ts_tablet_manager.cc:1428] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 07:03:13.216004 24299 heartbeater.cc:503] Master 127.15.250.190:35135 was elected leader, sending a full tablet report...
I20260808 07:03:13.216533 24589 raft_consensus.cc:348] T 20fa711728674c50a616400f2fc53ca1 P c24e263e9ede48a0bc8c1e84fb8245e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0383e7089e6448eb84566b41e2415e2e" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34401 } } peers { permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 } } peers { permanent_uuid: "dc14e3d81a4e4d44bf00163aaab40610" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 43635 } }
I20260808 07:03:13.216706 24589 raft_consensus.cc:374] T 20fa711728674c50a616400f2fc53ca1 P c24e263e9ede48a0bc8c1e84fb8245e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:03:13.216755 24589 raft_consensus.cc:729] T 20fa711728674c50a616400f2fc53ca1 P c24e263e9ede48a0bc8c1e84fb8245e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c24e263e9ede48a0bc8c1e84fb8245e7, State: Initialized, Role: FOLLOWER
I20260808 07:03:13.216881 24589 consensus_queue.cc:260] T 20fa711728674c50a616400f2fc53ca1 P c24e263e9ede48a0bc8c1e84fb8245e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0383e7089e6448eb84566b41e2415e2e" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34401 } } peers { permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 } } peers { permanent_uuid: "dc14e3d81a4e4d44bf00163aaab40610" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 43635 } }
I20260808 07:03:13.217146 24589 ts_tablet_manager.cc:1428] T 20fa711728674c50a616400f2fc53ca1 P c24e263e9ede48a0bc8c1e84fb8245e7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 07:03:13.217175 24571 heartbeater.cc:503] Master 127.15.250.190:35135 was elected leader, sending a full tablet report...
W20260808 07:03:13.231093 24438 tablet.cc:2369] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 07:03:13.275291 24572 tablet.cc:2369] T 20fa711728674c50a616400f2fc53ca1 P c24e263e9ede48a0bc8c1e84fb8245e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 07:03:13.301819 24300 tablet.cc:2369] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 07:03:13.462204 24592 raft_consensus.cc:482] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 07:03:13.462401 24592 raft_consensus.cc:504] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0383e7089e6448eb84566b41e2415e2e" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34401 } } peers { permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 } } peers { permanent_uuid: "dc14e3d81a4e4d44bf00163aaab40610" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 43635 } }
I20260808 07:03:13.462939 24592 leader_election.cc:288] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0383e7089e6448eb84566b41e2415e2e (127.15.250.130:34401), c24e263e9ede48a0bc8c1e84fb8245e7 (127.15.250.131:39239)
I20260808 07:03:13.467708 24389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fa711728674c50a616400f2fc53ca1" candidate_uuid: "dc14e3d81a4e4d44bf00163aaab40610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0383e7089e6448eb84566b41e2415e2e" is_pre_election: true
I20260808 07:03:13.467943 24389 raft_consensus.cc:2454] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc14e3d81a4e4d44bf00163aaab40610 in term 0.
I20260808 07:03:13.468206 24186 leader_election.cc:302] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0383e7089e6448eb84566b41e2415e2e, dc14e3d81a4e4d44bf00163aaab40610; no voters:
I20260808 07:03:13.468370 24592 raft_consensus.cc:2781] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 07:03:13.468420 24592 raft_consensus.cc:482] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 07:03:13.468447 24592 raft_consensus.cc:3037] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:13.468431 24524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fa711728674c50a616400f2fc53ca1" candidate_uuid: "dc14e3d81a4e4d44bf00163aaab40610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" is_pre_election: true
I20260808 07:03:13.468606 24524 raft_consensus.cc:2454] T 20fa711728674c50a616400f2fc53ca1 P c24e263e9ede48a0bc8c1e84fb8245e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc14e3d81a4e4d44bf00163aaab40610 in term 0.
I20260808 07:03:13.469348 24592 raft_consensus.cc:504] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0383e7089e6448eb84566b41e2415e2e" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34401 } } peers { permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 } } peers { permanent_uuid: "dc14e3d81a4e4d44bf00163aaab40610" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 43635 } }
I20260808 07:03:13.469604 24592 leader_election.cc:288] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [CANDIDATE]: Term 1 election: Requested vote from peers 0383e7089e6448eb84566b41e2415e2e (127.15.250.130:34401), c24e263e9ede48a0bc8c1e84fb8245e7 (127.15.250.131:39239)
I20260808 07:03:13.469703 24389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fa711728674c50a616400f2fc53ca1" candidate_uuid: "dc14e3d81a4e4d44bf00163aaab40610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0383e7089e6448eb84566b41e2415e2e"
I20260808 07:03:13.469712 24524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fa711728674c50a616400f2fc53ca1" candidate_uuid: "dc14e3d81a4e4d44bf00163aaab40610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7"
I20260808 07:03:13.469784 24389 raft_consensus.cc:3037] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:13.469796 24524 raft_consensus.cc:3037] T 20fa711728674c50a616400f2fc53ca1 P c24e263e9ede48a0bc8c1e84fb8245e7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:13.470472 24389 raft_consensus.cc:2454] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc14e3d81a4e4d44bf00163aaab40610 in term 1.
I20260808 07:03:13.470635 24186 leader_election.cc:302] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0383e7089e6448eb84566b41e2415e2e, dc14e3d81a4e4d44bf00163aaab40610; no voters:
I20260808 07:03:13.470741 24592 raft_consensus.cc:2781] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:13.470842 24524 raft_consensus.cc:2454] T 20fa711728674c50a616400f2fc53ca1 P c24e263e9ede48a0bc8c1e84fb8245e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc14e3d81a4e4d44bf00163aaab40610 in term 1.
I20260808 07:03:13.470916 24592 raft_consensus.cc:686] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [term 1 LEADER]: Becoming Leader. State: Replica: dc14e3d81a4e4d44bf00163aaab40610, State: Running, Role: LEADER
I20260808 07:03:13.471017 24592 consensus_queue.cc:237] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0383e7089e6448eb84566b41e2415e2e" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34401 } } peers { permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 } } peers { permanent_uuid: "dc14e3d81a4e4d44bf00163aaab40610" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 43635 } }
I20260808 07:03:13.471786 24108 catalog_manager.cc:5495] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc14e3d81a4e4d44bf00163aaab40610 (127.15.250.129). New cstate: current_term: 1 leader_uuid: "dc14e3d81a4e4d44bf00163aaab40610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0383e7089e6448eb84566b41e2415e2e" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34401 } } peers { permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 } } peers { permanent_uuid: "dc14e3d81a4e4d44bf00163aaab40610" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 43635 } } }
I20260808 07:03:13.487820 16362 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 07:03:13.488402 16362 ts_itest-base.cc:233] Waiting for 1 tablets on tserver dc14e3d81a4e4d44bf00163aaab40610 to finish bootstrapping
I20260808 07:03:13.492609 16362 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0383e7089e6448eb84566b41e2415e2e to finish bootstrapping
I20260808 07:03:13.496229 16362 ts_itest-base.cc:233] Waiting for 1 tablets on tserver c24e263e9ede48a0bc8c1e84fb8245e7 to finish bootstrapping
I20260808 07:03:13.509308 16362 cluster_itest_util.cc:256] Not converged past 1 yet: 0.0 0.0 1.1
I20260808 07:03:13.610785 16362 cluster_itest_util.cc:256] Not converged past 1 yet: 0.0 0.0 1.1
I20260808 07:03:13.812630 16362 cluster_itest_util.cc:256] Not converged past 1 yet: 0.0 0.0 1.1
I20260808 07:03:13.930375 24592 consensus_queue.cc:1035] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0383e7089e6448eb84566b41e2415e2e" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:03:13.932561 24596 consensus_queue.cc:1035] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:03:14.226720 16362 test_util.cc:268] Using random seed: -1224964591
I20260808 07:03:14.531309 16362 test_util.cc:268] Using random seed: -1224659998
I20260808 07:03:14.934553 16362 test_util.cc:268] Using random seed: -1224256756
I20260808 07:03:15.432333 24607 raft_consensus.cc:482] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dc14e3d81a4e4d44bf00163aaab40610)
I20260808 07:03:15.432451 24607 raft_consensus.cc:504] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0383e7089e6448eb84566b41e2415e2e" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34401 } } peers { permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 } } peers { permanent_uuid: "dc14e3d81a4e4d44bf00163aaab40610" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 43635 } }
I20260808 07:03:15.432834 24607 leader_election.cc:288] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c24e263e9ede48a0bc8c1e84fb8245e7 (127.15.250.131:39239), dc14e3d81a4e4d44bf00163aaab40610 (127.15.250.129:43635)
I20260808 07:03:15.433256 24610 raft_consensus.cc:482] T 20fa711728674c50a616400f2fc53ca1 P c24e263e9ede48a0bc8c1e84fb8245e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dc14e3d81a4e4d44bf00163aaab40610)
I20260808 07:03:15.433391 24610 raft_consensus.cc:504] T 20fa711728674c50a616400f2fc53ca1 P c24e263e9ede48a0bc8c1e84fb8245e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0383e7089e6448eb84566b41e2415e2e" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34401 } } peers { permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 } } peers { permanent_uuid: "dc14e3d81a4e4d44bf00163aaab40610" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 43635 } }
I20260808 07:03:15.433734 24610 leader_election.cc:288] T 20fa711728674c50a616400f2fc53ca1 P c24e263e9ede48a0bc8c1e84fb8245e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0383e7089e6448eb84566b41e2415e2e (127.15.250.130:34401), dc14e3d81a4e4d44bf00163aaab40610 (127.15.250.129:43635)
I20260808 07:03:15.437124 24524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fa711728674c50a616400f2fc53ca1" candidate_uuid: "0383e7089e6448eb84566b41e2415e2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" is_pre_election: true
I20260808 07:03:15.437223 24524 raft_consensus.cc:2454] T 20fa711728674c50a616400f2fc53ca1 P c24e263e9ede48a0bc8c1e84fb8245e7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0383e7089e6448eb84566b41e2415e2e in term 1.
I20260808 07:03:15.437398 24320 leader_election.cc:302] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0383e7089e6448eb84566b41e2415e2e, c24e263e9ede48a0bc8c1e84fb8245e7; no voters:
I20260808 07:03:15.437561 24607 raft_consensus.cc:2781] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 07:03:15.437602 24607 raft_consensus.cc:482] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader dc14e3d81a4e4d44bf00163aaab40610)
I20260808 07:03:15.437652 24607 raft_consensus.cc:3037] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [term 1 FOLLOWER]: Advancing to term 2
I20260808 07:03:15.437682 16362 test_util.cc:268] Using random seed: -1223753621
I20260808 07:03:15.437902 24389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fa711728674c50a616400f2fc53ca1" candidate_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0383e7089e6448eb84566b41e2415e2e" is_pre_election: true
I20260808 07:03:15.438390 24607 raft_consensus.cc:504] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0383e7089e6448eb84566b41e2415e2e" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34401 } } peers { permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 } } peers { permanent_uuid: "dc14e3d81a4e4d44bf00163aaab40610" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 43635 } }
I20260808 07:03:15.438506 24389 raft_consensus.cc:2379] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c24e263e9ede48a0bc8c1e84fb8245e7 in current term 2: Already voted for candidate 0383e7089e6448eb84566b41e2415e2e in this term.
I20260808 07:03:15.438532 24607 leader_election.cc:288] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [CANDIDATE]: Term 2 election: Requested vote from peers c24e263e9ede48a0bc8c1e84fb8245e7 (127.15.250.131:39239), dc14e3d81a4e4d44bf00163aaab40610 (127.15.250.129:43635)
I20260808 07:03:15.438688 24524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fa711728674c50a616400f2fc53ca1" candidate_uuid: "0383e7089e6448eb84566b41e2415e2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7"
I20260808 07:03:15.438767 24524 raft_consensus.cc:3037] T 20fa711728674c50a616400f2fc53ca1 P c24e263e9ede48a0bc8c1e84fb8245e7 [term 1 FOLLOWER]: Advancing to term 2
I20260808 07:03:15.439365 24524 raft_consensus.cc:2454] T 20fa711728674c50a616400f2fc53ca1 P c24e263e9ede48a0bc8c1e84fb8245e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0383e7089e6448eb84566b41e2415e2e in term 2.
I20260808 07:03:15.439555 24320 leader_election.cc:302] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0383e7089e6448eb84566b41e2415e2e, c24e263e9ede48a0bc8c1e84fb8245e7; no voters:
I20260808 07:03:15.439644 24607 raft_consensus.cc:2781] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [term 2 FOLLOWER]: Leader election won for term 2
I20260808 07:03:15.439814 24607 raft_consensus.cc:686] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [term 2 LEADER]: Becoming Leader. State: Replica: 0383e7089e6448eb84566b41e2415e2e, State: Running, Role: LEADER
I20260808 07:03:15.439910 24607 consensus_queue.cc:237] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0383e7089e6448eb84566b41e2415e2e" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34401 } } peers { permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 } } peers { permanent_uuid: "dc14e3d81a4e4d44bf00163aaab40610" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 43635 } }
I20260808 07:03:15.440636 24108 catalog_manager.cc:5495] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e reported cstate change: term changed from 1 to 2, leader changed from dc14e3d81a4e4d44bf00163aaab40610 (127.15.250.129) to 0383e7089e6448eb84566b41e2415e2e (127.15.250.130). New cstate: current_term: 2 leader_uuid: "0383e7089e6448eb84566b41e2415e2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0383e7089e6448eb84566b41e2415e2e" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34401 } } peers { permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 } } peers { permanent_uuid: "dc14e3d81a4e4d44bf00163aaab40610" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 43635 } } }
I20260808 07:03:15.941213 16362 test_util.cc:268] Using random seed: -1223250093
I20260808 07:03:15.947368 16362 test_workload.cc:401] TestWorkload: Skipping table creation because table TestTable already exists
I20260808 07:03:15.947687 16362 raft_consensus-itest-base.cc:268] Waiting until we've written at least 4MB...
I20260808 07:03:15.955662 24524 raft_consensus.cc:1261] T 20fa711728674c50a616400f2fc53ca1 P c24e263e9ede48a0bc8c1e84fb8245e7 [term 2 FOLLOWER]: Refusing update from remote peer 0383e7089e6448eb84566b41e2415e2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 07:03:15.956113 24607 consensus_queue.cc:1035] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 07:03:15.981463 24572 maintenance_manager.cc:419] P c24e263e9ede48a0bc8c1e84fb8245e7: Scheduling FlushMRSOp(20fa711728674c50a616400f2fc53ca1): 2498374 bytes log retention, and flush 4038921 bytes memory
I20260808 07:03:15.990083 24459 maintenance_manager.cc:643] P c24e263e9ede48a0bc8c1e84fb8245e7: FlushMRSOp(20fa711728674c50a616400f2fc53ca1) complete. Timing: real 0.008s user 0.001s sys 0.005s Metrics: {"bytes_written":135869,"cfile_init":1,"compiler_manager_pool.queue_time_us":192,"dirs.queue_time_us":489,"dirs.run_cpu_time_us":141,"dirs.run_wall_time_us":1127,"drs_written":1,"lbm_read_time_us":27,"lbm_reads_lt_1ms":4,"lbm_write_time_us":339,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":22,"thread_start_us":252,"threads_started":2}
I20260808 07:03:15.990368 24572 maintenance_manager.cc:419] P c24e263e9ede48a0bc8c1e84fb8245e7: Scheduling LogGCOp(20fa711728674c50a616400f2fc53ca1): free 1709421 bytes of WAL
I20260808 07:03:15.990504 24459 log_reader.cc:386] T 20fa711728674c50a616400f2fc53ca1: removed 2 log segments from log reader
I20260808 07:03:15.990548 24459 log.cc:1076] T 20fa711728674c50a616400f2fc53ca1 P c24e263e9ede48a0bc8c1e84fb8245e7: Deleting log segment in path: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal/wals/20fa711728674c50a616400f2fc53ca1/wal-000000001 (ops 1-8)
I20260808 07:03:15.990576 24459 log.cc:1076] T 20fa711728674c50a616400f2fc53ca1 P c24e263e9ede48a0bc8c1e84fb8245e7: Deleting log segment in path: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal/wals/20fa711728674c50a616400f2fc53ca1/wal-000000002 (ops 9-15)
I20260808 07:03:15.991299 24459 maintenance_manager.cc:643] P c24e263e9ede48a0bc8c1e84fb8245e7: LogGCOp(20fa711728674c50a616400f2fc53ca1) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {}
I20260808 07:03:15.991434 24572 maintenance_manager.cc:419] P c24e263e9ede48a0bc8c1e84fb8245e7: Scheduling UndoDeltaBlockGCOp(20fa711728674c50a616400f2fc53ca1): 378 bytes on disk
I20260808 07:03:15.991679 24459 maintenance_manager.cc:643] P c24e263e9ede48a0bc8c1e84fb8245e7: UndoDeltaBlockGCOp(20fa711728674c50a616400f2fc53ca1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":27,"lbm_reads_lt_1ms":4}
I20260808 07:03:16.000309 16362 raft_consensus-itest-base.cc:274] Waiting for log GC on 0383e7089e6448eb84566b41e2415e2e
I20260808 07:03:16.036753 24438 maintenance_manager.cc:419] P 0383e7089e6448eb84566b41e2415e2e: Scheduling FlushMRSOp(20fa711728674c50a616400f2fc53ca1): 4602350 bytes log retention, and flush 6119689 bytes memory
I20260808 07:03:16.045341 24323 maintenance_manager.cc:643] P 0383e7089e6448eb84566b41e2415e2e: FlushMRSOp(20fa711728674c50a616400f2fc53ca1) complete. Timing: real 0.008s user 0.001s sys 0.007s Metrics: {"bytes_written":136001,"cfile_init":1,"compiler_manager_pool.queue_time_us":171,"dirs.queue_time_us":170,"dirs.run_cpu_time_us":157,"dirs.run_wall_time_us":1221,"drs_written":1,"lbm_read_time_us":28,"lbm_reads_lt_1ms":4,"lbm_write_time_us":422,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":38,"thread_start_us":236,"threads_started":2}
I20260808 07:03:16.045590 24438 maintenance_manager.cc:419] P 0383e7089e6448eb84566b41e2415e2e: Scheduling LogGCOp(20fa711728674c50a616400f2fc53ca1): free 2761362 bytes of WAL
I20260808 07:03:16.045699 24323 log_reader.cc:386] T 20fa711728674c50a616400f2fc53ca1: removed 3 log segments from log reader
I20260808 07:03:16.045764 24323 log.cc:1076] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e: Deleting log segment in path: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/wals/20fa711728674c50a616400f2fc53ca1/wal-000000001 (ops 1-9)
I20260808 07:03:16.045792 24323 log.cc:1076] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e: Deleting log segment in path: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/wals/20fa711728674c50a616400f2fc53ca1/wal-000000002 (ops 10-16)
I20260808 07:03:16.045853 24323 log.cc:1076] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e: Deleting log segment in path: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/wals/20fa711728674c50a616400f2fc53ca1/wal-000000003 (ops 17-23)
I20260808 07:03:16.046798 24323 maintenance_manager.cc:643] P 0383e7089e6448eb84566b41e2415e2e: LogGCOp(20fa711728674c50a616400f2fc53ca1) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20260808 07:03:16.046960 24438 maintenance_manager.cc:419] P 0383e7089e6448eb84566b41e2415e2e: Scheduling UndoDeltaBlockGCOp(20fa711728674c50a616400f2fc53ca1): 558 bytes on disk
I20260808 07:03:16.047186 24323 maintenance_manager.cc:643] P 0383e7089e6448eb84566b41e2415e2e: UndoDeltaBlockGCOp(20fa711728674c50a616400f2fc53ca1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":27,"lbm_reads_lt_1ms":4}
I20260808 07:03:16.051757 16362 raft_consensus-itest-base.cc:279] Log GC complete on 0383e7089e6448eb84566b41e2415e2e
I20260808 07:03:16.091984 24572 maintenance_manager.cc:419] P c24e263e9ede48a0bc8c1e84fb8245e7: Scheduling LogGCOp(20fa711728674c50a616400f2fc53ca1): free 788953 bytes of WAL
I20260808 07:03:16.092140 24459 log_reader.cc:386] T 20fa711728674c50a616400f2fc53ca1: removed 1 log segments from log reader
I20260808 07:03:16.092163 24459 log.cc:1076] T 20fa711728674c50a616400f2fc53ca1 P c24e263e9ede48a0bc8c1e84fb8245e7: Deleting log segment in path: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal/wals/20fa711728674c50a616400f2fc53ca1/wal-000000003 (ops 16-21)
I20260808 07:03:16.092536 24459 maintenance_manager.cc:643] P c24e263e9ede48a0bc8c1e84fb8245e7: LogGCOp(20fa711728674c50a616400f2fc53ca1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260808 07:03:16.092689 24572 maintenance_manager.cc:419] P c24e263e9ede48a0bc8c1e84fb8245e7: Scheduling FlushMRSOp(20fa711728674c50a616400f2fc53ca1): 1577874 bytes log retention, and flush 2998537 bytes memory
I20260808 07:03:16.096191 24459 maintenance_manager.cc:643] P c24e263e9ede48a0bc8c1e84fb8245e7: FlushMRSOp(20fa711728674c50a616400f2fc53ca1) complete. Timing: real 0.003s user 0.000s sys 0.002s Metrics: {"bytes_written":135803,"cfile_init":1,"dirs.queue_time_us":61,"drs_written":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4,"lbm_write_time_us":338,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":16}
I20260808 07:03:16.096408 24572 maintenance_manager.cc:419] P c24e263e9ede48a0bc8c1e84fb8245e7: Scheduling UndoDeltaBlockGCOp(20fa711728674c50a616400f2fc53ca1): 311 bytes on disk
I20260808 07:03:16.096563 24459 maintenance_manager.cc:643] P c24e263e9ede48a0bc8c1e84fb8245e7: UndoDeltaBlockGCOp(20fa711728674c50a616400f2fc53ca1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":24,"lbm_reads_lt_1ms":4}
I20260808 07:03:16.096817 24572 maintenance_manager.cc:419] P c24e263e9ede48a0bc8c1e84fb8245e7: Scheduling CompactRowSetsOp(20fa711728674c50a616400f2fc53ca1): perf score=0.987519
I20260808 07:03:16.101557 24459 maintenance_manager.cc:643] P c24e263e9ede48a0bc8c1e84fb8245e7: CompactRowSetsOp(20fa711728674c50a616400f2fc53ca1) complete. Timing: real 0.005s user 0.004s sys 0.000s Metrics: {"bytes_written":136001,"cfile_cache_hit":10,"cfile_cache_hit_bytes":262392,"cfile_cache_miss":26,"cfile_cache_miss_bytes":264318,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":432,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":1585,"drs_written":1,"lbm_read_time_us":230,"lbm_reads_lt_1ms":46,"lbm_write_time_us":297,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":284,"rows_written":38,"thread_start_us":323,"threads_started":5}
I20260808 07:03:16.101751 24572 maintenance_manager.cc:419] P c24e263e9ede48a0bc8c1e84fb8245e7: Scheduling UndoDeltaBlockGCOp(20fa711728674c50a616400f2fc53ca1): 558 bytes on disk
I20260808 07:03:16.101929 24459 maintenance_manager.cc:643] P c24e263e9ede48a0bc8c1e84fb8245e7: UndoDeltaBlockGCOp(20fa711728674c50a616400f2fc53ca1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4}
W20260808 07:03:17.163915 24458 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.15.250.129:43635 for RPC RequestConsensusVote timed out after 1.728s (ON_OUTBOUND_QUEUE)
W20260808 07:03:17.350033 24322 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.15.250.129:43635 for RPC RequestConsensusVote timed out after 1.910s (ON_OUTBOUND_QUEUE)
I20260808 07:03:18.052600 16362 raft_consensus-itest-base.cc:295] Servers converged with original term 2
I20260808 07:03:18.052698 16362 raft_consensus-itest-base.cc:300] Resuming dc14e3d81a4e4d44bf00163aaab40610
I20260808 07:03:18.054198 24524 raft_consensus.cc:1226] T 20fa711728674c50a616400f2fc53ca1 P c24e263e9ede48a0bc8c1e84fb8245e7 [term 2 FOLLOWER]: Rejecting Update request from peer dc14e3d81a4e4d44bf00163aaab40610 for earlier term 1. Current term is 2. Ops: []
I20260808 07:03:18.054579 24596 consensus_queue.cc:1046] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 40, Time since last communication: 0.000s
I20260808 07:03:18.054718 24596 raft_consensus.cc:3032] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 07:03:18.054746 24596 raft_consensus.cc:729] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dc14e3d81a4e4d44bf00163aaab40610, State: Running, Role: LEADER
I20260808 07:03:18.054792 24596 consensus_queue.cc:260] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0383e7089e6448eb84566b41e2415e2e" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34401 } } peers { permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 } } peers { permanent_uuid: "dc14e3d81a4e4d44bf00163aaab40610" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 43635 } }
I20260808 07:03:18.054955 24596 raft_consensus.cc:3037] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [term 1 FOLLOWER]: Advancing to term 2
I20260808 07:03:18.056131 24388 raft_consensus.cc:1226] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [term 2 LEADER]: Rejecting Update request from peer dc14e3d81a4e4d44bf00163aaab40610 for earlier term 1. Current term is 2. Ops: []
I20260808 07:03:18.058298 24248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fa711728674c50a616400f2fc53ca1" candidate_uuid: "0383e7089e6448eb84566b41e2415e2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dc14e3d81a4e4d44bf00163aaab40610" is_pre_election: true
I20260808 07:03:18.058341 24247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fa711728674c50a616400f2fc53ca1" candidate_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dc14e3d81a4e4d44bf00163aaab40610" is_pre_election: true
I20260808 07:03:18.058301 24249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fa711728674c50a616400f2fc53ca1" candidate_uuid: "0383e7089e6448eb84566b41e2415e2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dc14e3d81a4e4d44bf00163aaab40610"
I20260808 07:03:18.058393 24246 raft_consensus.cc:1261] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [term 2 FOLLOWER]: Refusing update from remote peer 0383e7089e6448eb84566b41e2415e2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 07:03:18.058606 24458 leader_election.cc:302] T 20fa711728674c50a616400f2fc53ca1 P c24e263e9ede48a0bc8c1e84fb8245e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c24e263e9ede48a0bc8c1e84fb8245e7; no voters: 0383e7089e6448eb84566b41e2415e2e, dc14e3d81a4e4d44bf00163aaab40610
I20260808 07:03:18.058843 24664 raft_consensus.cc:2726] T 20fa711728674c50a616400f2fc53ca1 P c24e263e9ede48a0bc8c1e84fb8245e7 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:03:18.058828 24665 consensus_queue.cc:1035] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc14e3d81a4e4d44bf00163aaab40610" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 07:03:18.059077 24665 consensus_queue.cc:740] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [LEADER]: The logs necessary to catch up peer dc14e3d81a4e4d44bf00163aaab40610 have been garbage collected. The follower will never be able to catch up (Not found: failed to read ops 2..33: Segment 1 which contained index 2 has been GCed)
I20260808 07:03:18.059247 24665 raft_consensus.cc:998] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e: Attempting to remove follower dc14e3d81a4e4d44bf00163aaab40610 from the Raft config. Reason: The logs necessary to catch up peer dc14e3d81a4e4d44bf00163aaab40610 have been garbage collected. The replica will never be able to catch up
I20260808 07:03:18.059530 24665 consensus_queue.cc:237] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 2.40, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "0383e7089e6448eb84566b41e2415e2e" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34401 } } peers { permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 } }
I20260808 07:03:18.060058 24524 raft_consensus.cc:1261] T 20fa711728674c50a616400f2fc53ca1 P c24e263e9ede48a0bc8c1e84fb8245e7 [term 2 FOLLOWER]: Refusing update from remote peer 0383e7089e6448eb84566b41e2415e2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 40. Preceding OpId from leader: term: 2 index: 41. (index mismatch)
I20260808 07:03:18.060335 24667 consensus_queue.cc:1035] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20260808 07:03:18.061272 24665 raft_consensus.cc:2932] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [term 2 LEADER]: Committing config change with OpId 2.41: config changed from index -1 to 41, VOTER dc14e3d81a4e4d44bf00163aaab40610 (127.15.250.129) evicted. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "0383e7089e6448eb84566b41e2415e2e" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34401 } } peers { permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 } } }
I20260808 07:03:18.061403 24524 raft_consensus.cc:2932] T 20fa711728674c50a616400f2fc53ca1 P c24e263e9ede48a0bc8c1e84fb8245e7 [term 2 FOLLOWER]: Committing config change with OpId 2.41: config changed from index -1 to 41, VOTER dc14e3d81a4e4d44bf00163aaab40610 (127.15.250.129) evicted. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "0383e7089e6448eb84566b41e2415e2e" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34401 } } peers { permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 } } }
I20260808 07:03:18.062728 24108 catalog_manager.cc:5495] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e reported cstate change: config changed from index -1 to 41, VOTER dc14e3d81a4e4d44bf00163aaab40610 (127.15.250.129) evicted. New cstate: current_term: 2 leader_uuid: "0383e7089e6448eb84566b41e2415e2e" committed_config { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "0383e7089e6448eb84566b41e2415e2e" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34401 } } peers { permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 } } }
I20260808 07:03:18.064080 24233 tablet_service.cc:1514] Processing DeleteTablet for tablet 20fa711728674c50a616400f2fc53ca1 with delete_type TABLET_DATA_TOMBSTONED (TS dc14e3d81a4e4d44bf00163aaab40610 not found in new config with opid_index 41) from {username='slave'} at 127.0.0.1:41600
I20260808 07:03:18.064405 24674 tablet_replica.cc:330] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: stopping tablet replica
I20260808 07:03:18.064602 24674 raft_consensus.cc:2229] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 07:03:18.064702 24674 raft_consensus.cc:2258] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 07:03:18.065019 24674 ts_tablet_manager.cc:1905] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 07:03:18.066859 24674 ts_tablet_manager.cc:1918] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260808 07:03:18.066927 24674 log.cc:1196] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: Deleting WAL directory at /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/wals/20fa711728674c50a616400f2fc53ca1
I20260808 07:03:18.067359 24095 catalog_manager.cc:4841] TS dc14e3d81a4e4d44bf00163aaab40610 (127.15.250.129:43635): tablet 20fa711728674c50a616400f2fc53ca1 (table TestTable [id=5209b2a3bff141b99389ae23512a4cdb]) successfully deleted
I20260808 07:03:18.071965 24293 generic_service.cc:175] {username='slave'} at 127.0.0.1:41604 changed flags via RPC: tablet_copy_fault_crash_on_fetch_all from '0' to '1.0'
I20260808 07:03:18.072813 24388 consensus_queue.cc:237] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 2.41, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "0383e7089e6448eb84566b41e2415e2e" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34401 } } peers { permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 } } peers { permanent_uuid: "dc14e3d81a4e4d44bf00163aaab40610" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 43635 } attrs { } }
I20260808 07:03:18.073449 24524 raft_consensus.cc:1261] T 20fa711728674c50a616400f2fc53ca1 P c24e263e9ede48a0bc8c1e84fb8245e7 [term 2 FOLLOWER]: Refusing update from remote peer 0383e7089e6448eb84566b41e2415e2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 41. Preceding OpId from leader: term: 2 index: 42. (index mismatch)
W20260808 07:03:18.073702 24322 consensus_peers.cc:487] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e -> Peer dc14e3d81a4e4d44bf00163aaab40610 (127.15.250.129:43635): Couldn't send request to peer dc14e3d81a4e4d44bf00163aaab40610. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260808 07:03:18.073709 24665 consensus_queue.cc:1035] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20260808 07:03:18.074709 24667 raft_consensus.cc:2932] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [term 2 LEADER]: Committing config change with OpId 2.42: config changed from index 41 to 42, VOTER dc14e3d81a4e4d44bf00163aaab40610 (127.15.250.129) added. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "0383e7089e6448eb84566b41e2415e2e" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34401 } } peers { permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 } } peers { permanent_uuid: "dc14e3d81a4e4d44bf00163aaab40610" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 43635 } attrs { } } }
I20260808 07:03:18.074868 24524 raft_consensus.cc:2932] T 20fa711728674c50a616400f2fc53ca1 P c24e263e9ede48a0bc8c1e84fb8245e7 [term 2 FOLLOWER]: Committing config change with OpId 2.42: config changed from index 41 to 42, VOTER dc14e3d81a4e4d44bf00163aaab40610 (127.15.250.129) added. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "0383e7089e6448eb84566b41e2415e2e" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34401 } } peers { permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 } } peers { permanent_uuid: "dc14e3d81a4e4d44bf00163aaab40610" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 43635 } attrs { } } }
I20260808 07:03:18.076193 24108 catalog_manager.cc:5495] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e reported cstate change: config changed from index 41 to 42, VOTER dc14e3d81a4e4d44bf00163aaab40610 (127.15.250.129) added. New cstate: current_term: 2 leader_uuid: "0383e7089e6448eb84566b41e2415e2e" committed_config { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "0383e7089e6448eb84566b41e2415e2e" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34401 } } peers { permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 } } peers { permanent_uuid: "dc14e3d81a4e4d44bf00163aaab40610" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 43635 } attrs { } } }
I20260808 07:03:18.656360 24676 ts_tablet_manager.cc:927] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: Initiating tablet copy from peer 0383e7089e6448eb84566b41e2415e2e (127.15.250.130:34401)
I20260808 07:03:18.656562 24676 tablet_copy_client.cc:316] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: tablet copy: Beginning tablet copy session from remote peer at address 127.15.250.130:34401
I20260808 07:03:18.656932 24410 tablet_copy_service.cc:140] P 0383e7089e6448eb84566b41e2415e2e: Received BeginTabletCopySession request for tablet 20fa711728674c50a616400f2fc53ca1 from peer dc14e3d81a4e4d44bf00163aaab40610 ({username='slave'} at 127.15.250.129:57581)
I20260808 07:03:18.657037 24410 tablet_copy_service.cc:161] P 0383e7089e6448eb84566b41e2415e2e: Beginning new tablet copy session on tablet 20fa711728674c50a616400f2fc53ca1 from peer dc14e3d81a4e4d44bf00163aaab40610 at {username='slave'} at 127.15.250.129:57581: session id = dc14e3d81a4e4d44bf00163aaab40610-20fa711728674c50a616400f2fc53ca1
I20260808 07:03:18.657754 24410 tablet_copy_source_session.cc:205] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e: Tablet Copy: opened 5 blocks and 3 log segments
I20260808 07:03:18.658473 24676 ts_tablet_manager.cc:1905] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260808 07:03:18.660115 24676 ts_tablet_manager.cc:1918] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1
I20260808 07:03:18.660245 24676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20fa711728674c50a616400f2fc53ca1. 1 dirs total, 0 dirs full, 0 dirs failed
W20260808 07:03:18.661033 24676 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260808 07:03:18.661082 24676 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260808 07:03:18.661134 24676 fault_injection.cc:59] injecting fault for kudu (pid 24169): FLAGS_tablet_copy_fault_crash_on_fetch_all (process will exit)
I20260808 07:03:18.737800 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 24302
I20260808 07:03:18.745189 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 24440
I20260808 07:03:18.751940 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 24075
I20260808 07:03:18.758801 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.250.190:35135
--webserver_interface=127.15.250.190
--webserver_port=36173
--builtin_ntp_servers=127.15.250.148:41221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.250.190:35135
--master_add_server_when_underreplicated=false
--raft_prepare_replacement_before_eviction=false with env {}
W20260808 07:03:18.845036 24678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:18.845265 24678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:18.845288 24678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:18.846580 24678 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260808 07:03:18.846634 24678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:03:18.846647 24678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:18.846660 24678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:03:18.846714 24678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:03:18.848160 24678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:41221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
--master_add_server_when_underreplicated=false
--ipki_ca_key_size=768
--master_addresses=127.15.250.190:35135
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.250.190:35135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.250.190
--webserver_port=36173
--never_fsync=true
--heap_profile_path=/tmp/kudu.24678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:18.848443 24678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:18.848733 24678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:18.854038 24683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:18.854116 24684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:18.854135 24686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:18.854171 24678 server_base.cc:1034] running on GCE node
I20260808 07:03:18.854664 24678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:18.854976 24678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:18.856124 24678 hybrid_clock.cc:648] HybridClock initialized: now 1786172598856103 us; error 36 us; skew 500 ppm
I20260808 07:03:18.862712 24678 webserver.cc:460] Webserver started at http://127.15.250.190:36173/ using document root <none> and password file <none>
I20260808 07:03:18.863107 24678 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:18.863202 24678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:18.865028 24678 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:03:18.865799 24692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:18.866078 24678 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:03:18.866194 24678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
uuid: "74682779344e47ecbb8b8e06a774afc5"
format_stamp: "Formatted at 2026-08-08 07:03:12 on dist-test-slave-0b58"
I20260808 07:03:18.866621 24678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:18.898438 24678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:18.898820 24678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:18.899021 24678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:18.903432 24678 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.190:35135
I20260808 07:03:18.903469 24745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.190:35135 every 8 connection(s)
I20260808 07:03:18.903869 24678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260808 07:03:18.905105 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 24678
I20260808 07:03:18.905504 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.130:34401
--local_ip_for_outbound_sockets=127.15.250.130
--tserver_master_addrs=127.15.250.190:35135
--webserver_port=44221
--webserver_interface=127.15.250.130
--builtin_ntp_servers=127.15.250.148:41221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20260808 07:03:18.906318 24746 sys_catalog.cc:263] Verifying existing consensus state
I20260808 07:03:18.907470 24746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5: Bootstrap starting.
I20260808 07:03:18.915230 24746 log.cc:824] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:18.917254 24746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=5 ignored=0} mutations{seen=5 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 07:03:18.917567 24746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5: Bootstrap complete.
I20260808 07:03:18.919953 24746 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74682779344e47ecbb8b8e06a774afc5" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 35135 } }
I20260808 07:03:18.920220 24746 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74682779344e47ecbb8b8e06a774afc5, State: Initialized, Role: FOLLOWER
I20260808 07:03:18.920442 24746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74682779344e47ecbb8b8e06a774afc5" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 35135 } }
I20260808 07:03:18.920543 24746 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:03:18.920599 24746 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:03:18.920655 24746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 07:03:18.921609 24746 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74682779344e47ecbb8b8e06a774afc5" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 35135 } }
I20260808 07:03:18.921736 24746 leader_election.cc:302] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74682779344e47ecbb8b8e06a774afc5; no voters:
I20260808 07:03:18.921947 24746 leader_election.cc:288] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 07:03:18.921988 24751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 07:03:18.922102 24751 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [term 2 LEADER]: Becoming Leader. State: Replica: 74682779344e47ecbb8b8e06a774afc5, State: Running, Role: LEADER
I20260808 07:03:18.922237 24746 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:03:18.922260 24751 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74682779344e47ecbb8b8e06a774afc5" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 35135 } }
I20260808 07:03:18.922955 24753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74682779344e47ecbb8b8e06a774afc5. Latest consensus state: current_term: 2 leader_uuid: "74682779344e47ecbb8b8e06a774afc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74682779344e47ecbb8b8e06a774afc5" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 35135 } } }
I20260808 07:03:18.922963 24752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "74682779344e47ecbb8b8e06a774afc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74682779344e47ecbb8b8e06a774afc5" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 35135 } } }
I20260808 07:03:18.923053 24753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [sys.catalog]: This master's current role is: LEADER
I20260808 07:03:18.923081 24752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5 [sys.catalog]: This master's current role is: LEADER
I20260808 07:03:18.923285 24758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:03:18.924064 24758 catalog_manager.cc:669] Loaded metadata for table TestTable [id=5209b2a3bff141b99389ae23512a4cdb]
I20260808 07:03:18.924532 24758 tablet_loader.cc:96] loaded metadata for tablet 20fa711728674c50a616400f2fc53ca1 (table TestTable [id=5209b2a3bff141b99389ae23512a4cdb])
I20260808 07:03:18.924618 24758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:03:18.924736 24758 catalog_manager.cc:1259] Loaded cluster ID: 9d7ddfb7f09f44cca0fa054921cee053
I20260808 07:03:18.924795 24758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:03:18.925804 24758 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:03:18.926004 24758 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 74682779344e47ecbb8b8e06a774afc5: Loaded TSK: 0
I20260808 07:03:18.926239 24758 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 07:03:19.000669 24749 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20260808 07:03:19.000952 24749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:19.000993 24749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:19.001029 24749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:19.002365 24749 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20260808 07:03:19.002439 24749 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20260808 07:03:19.002463 24749 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260808 07:03:19.002487 24749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:19.002534 24749 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20260808 07:03:19.002563 24749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.130
I20260808 07:03:19.004015 24749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:41221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.130:34401
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.250.130
--webserver_port=44221
--tserver_master_addrs=127.15.250.190:35135
--never_fsync=true
--heap_profile_path=/tmp/kudu.24749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.130
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:19.004360 24749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:19.004706 24749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:19.010108 24775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:19.010172 24779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:19.010108 24776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:19.010478 24749 server_base.cc:1034] running on GCE node
I20260808 07:03:19.010800 24749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:19.011122 24749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:19.012327 24749 hybrid_clock.cc:648] HybridClock initialized: now 1786172599012308 us; error 33 us; skew 500 ppm
I20260808 07:03:19.014003 24749 webserver.cc:460] Webserver started at http://127.15.250.130:44221/ using document root <none> and password file <none>
I20260808 07:03:19.014307 24749 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:19.014400 24749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:19.016328 24749 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 07:03:19.018667 24787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:19.018885 24749 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 07:03:19.019003 24749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "0383e7089e6448eb84566b41e2415e2e"
format_stamp: "Formatted at 2026-08-08 07:03:12 on dist-test-slave-0b58"
I20260808 07:03:19.019398 24749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 5
Total live bytes: 136559
Total live bytes (after alignment): 155648
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:19.029093 24749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:19.029429 24749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:19.029588 24749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:19.029834 24749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:03:19.030349 24794 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 07:03:19.032474 24749 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 07:03:19.032567 24749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260808 07:03:19.032624 24749 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 07:03:19.033305 24749 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 07:03:19.033370 24749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260808 07:03:19.033437 24794 tablet_bootstrap.cc:492] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e: Bootstrap starting.
I20260808 07:03:19.038630 24794 log.cc:824] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:19.040498 24749 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.130:34401
I20260808 07:03:19.040536 24904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.130:34401 every 8 connection(s)
I20260808 07:03:19.040918 24749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260808 07:03:19.042021 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 24749
I20260808 07:03:19.042407 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.129:43635
--local_ip_for_outbound_sockets=127.15.250.129
--tserver_master_addrs=127.15.250.190:35135
--webserver_port=35735
--webserver_interface=127.15.250.129
--builtin_ntp_servers=127.15.250.148:41221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20260808 07:03:19.043293 24794 tablet_bootstrap.cc:492] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e: Bootstrap replayed 1/3 log segments. Stats: ops{read=7 overwritten=0 applied=5 ignored=5} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=3. Pending: 2 replicates
I20260808 07:03:19.045738 24794 tablet_bootstrap.cc:492] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e: Bootstrap replayed 2/3 log segments. Stats: ops{read=14 overwritten=0 applied=13 ignored=13} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=3. Pending: 1 replicates
I20260808 07:03:19.049750 24905 heartbeater.cc:347] Connected to a master server at 127.15.250.190:35135
I20260808 07:03:19.049880 24905 heartbeater.cc:464] Registering TS with master...
I20260808 07:03:19.050063 24905 heartbeater.cc:511] Master 127.15.250.190:35135 requested a full tablet report, sending...
I20260808 07:03:19.050734 24710 ts_manager.cc:194] Registered new tserver with Master: 0383e7089e6448eb84566b41e2415e2e (127.15.250.130:34401)
I20260808 07:03:19.051451 24710 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.130:58515
I20260808 07:03:19.053426 24794 tablet_bootstrap.cc:492] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e: Bootstrap replayed 3/3 log segments. Stats: ops{read=19 overwritten=0 applied=19 ignored=17} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=3. Pending: 0 replicates
I20260808 07:03:19.053836 24794 tablet_bootstrap.cc:492] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e: Bootstrap complete.
I20260808 07:03:19.054497 24794 ts_tablet_manager.cc:1397] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.008s
I20260808 07:03:19.055505 24794 raft_consensus.cc:348] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "0383e7089e6448eb84566b41e2415e2e" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34401 } } peers { permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 } } peers { permanent_uuid: "dc14e3d81a4e4d44bf00163aaab40610" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 43635 } attrs { } }
I20260808 07:03:19.055814 24794 raft_consensus.cc:729] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0383e7089e6448eb84566b41e2415e2e, State: Initialized, Role: FOLLOWER
I20260808 07:03:19.056000 24794 consensus_queue.cc:260] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 42, Last appended: 2.42, Last appended by leader: 42, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "0383e7089e6448eb84566b41e2415e2e" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34401 } } peers { permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 } } peers { permanent_uuid: "dc14e3d81a4e4d44bf00163aaab40610" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 43635 } attrs { } }
I20260808 07:03:19.056387 24794 ts_tablet_manager.cc:1428] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
I20260808 07:03:19.056527 24905 heartbeater.cc:503] Master 127.15.250.190:35135 was elected leader, sending a full tablet report...
W20260808 07:03:19.138787 24909 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20260808 07:03:19.139014 24909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:19.139031 24909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:19.139050 24909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:19.140318 24909 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20260808 07:03:19.140360 24909 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20260808 07:03:19.140376 24909 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260808 07:03:19.140391 24909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:19.140414 24909 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20260808 07:03:19.140456 24909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.129
I20260808 07:03:19.141690 24906 maintenance_manager.cc:419] P 0383e7089e6448eb84566b41e2415e2e: Scheduling UndoDeltaBlockGCOp(20fa711728674c50a616400f2fc53ca1): 558 bytes on disk
I20260808 07:03:19.141889 24909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:41221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.129:43635
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.250.129
--webserver_port=35735
--tserver_master_addrs=127.15.250.190:35135
--never_fsync=true
--heap_profile_path=/tmp/kudu.24909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.129
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:19.142191 24909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:19.142180 24792 maintenance_manager.cc:643] P 0383e7089e6448eb84566b41e2415e2e: UndoDeltaBlockGCOp(20fa711728674c50a616400f2fc53ca1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":30,"lbm_reads_lt_1ms":4}
I20260808 07:03:19.142572 24909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:19.147920 24915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:19.147929 24918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:19.147993 24909 server_base.cc:1034] running on GCE node
W20260808 07:03:19.147929 24916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:19.148420 24909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:19.148643 24909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:19.149786 24909 hybrid_clock.cc:648] HybridClock initialized: now 1786172599149774 us; error 30 us; skew 500 ppm
I20260808 07:03:19.151297 24909 webserver.cc:460] Webserver started at http://127.15.250.129:35735/ using document root <none> and password file <none>
I20260808 07:03:19.151568 24909 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:19.151666 24909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:19.153157 24909 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 07:03:19.153887 24924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:19.154078 24909 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 07:03:19.154186 24909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "dc14e3d81a4e4d44bf00163aaab40610"
format_stamp: "Formatted at 2026-08-08 07:03:12 on dist-test-slave-0b58"
I20260808 07:03:19.154582 24909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:19.188656 24909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:19.189011 24909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:19.189177 24909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:19.189440 24909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:03:19.189967 24931 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260808 07:03:19.191440 24931 ts_tablet_manager.cc:1852] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_COPYING
I20260808 07:03:19.192206 24931 ts_tablet_manager.cc:1905] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 07:03:19.194406 24931 ts_tablet_manager.cc:1918] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260808 07:03:19.194758 24909 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260808 07:03:19.194819 24909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260808 07:03:19.194851 24909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:03:19.194865 24909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:19.202087 24909 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.129:43635
I20260808 07:03:19.202178 25040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.129:43635 every 8 connection(s)
I20260808 07:03:19.202541 24909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260808 07:03:19.208863 25041 heartbeater.cc:347] Connected to a master server at 127.15.250.190:35135
I20260808 07:03:19.208968 25041 heartbeater.cc:464] Registering TS with master...
I20260808 07:03:19.209144 25041 heartbeater.cc:511] Master 127.15.250.190:35135 requested a full tablet report, sending...
I20260808 07:03:19.209664 24710 ts_manager.cc:194] Registered new tserver with Master: dc14e3d81a4e4d44bf00163aaab40610 (127.15.250.129:43635)
I20260808 07:03:19.210079 24710 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.129:40053
I20260808 07:03:19.210624 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 24909
I20260808 07:03:19.210821 16362 test_util.cc:268] Using random seed: -1219980475
I20260808 07:03:19.217621 16362 test_workload.cc:401] TestWorkload: Skipping table creation because table TestTable already exists
W20260808 07:03:19.227182 24814 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.227187 24816 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.227182 24813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.227187 24815 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.229204 25046 meta_cache.cc:302] tablet 20fa711728674c50a616400f2fc53ca1: replica c24e263e9ede48a0bc8c1e84fb8245e7 (127.15.250.131:39239) has failed: Network error: Client connection negotiation failed: client connection to 127.15.250.131:39239: connect: Connection refused (error 111)
W20260808 07:03:19.236514 24815 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.246887 24815 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.254117 24815 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.254117 24813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.260423 24815 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.267805 24815 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.277452 24815 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.277556 24813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.286880 24813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.294410 24813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.304770 24813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.314815 24813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.328436 24813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.337914 24813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.346845 24813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.362476 24813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.376183 24813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.389894 24813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.401302 24813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.414943 24813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.433565 24813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.453007 24813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.453037 24815 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.474486 24815 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.486691 24815 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.508531 24815 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.531149 24815 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.557780 24815 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.577600 24815 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.603025 24815 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.625283 24815 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.657595 24815 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.674220 24815 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.692746 24815 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.720151 24815 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.747236 24815 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.770713 24815 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.788968 24815 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.822572 24815 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.852742 24815 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.875850 24815 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.896472 24815 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.928310 24815 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.965164 24813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:19.965164 24815 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:20.005229 24815 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:20.005450 24813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:20.044195 24813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:20.066164 24813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:20.095688 24813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:20.129603 24813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:20.165107 24813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:20.193888 24813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
I20260808 07:03:20.211028 25041 heartbeater.cc:503] Master 127.15.250.190:35135 was elected leader, sending a full tablet report...
W20260808 07:03:20.233362 24813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:20.238330 25046 meta_cache.cc:302] tablet 20fa711728674c50a616400f2fc53ca1: replica c24e263e9ede48a0bc8c1e84fb8245e7 (127.15.250.131:39239) has failed: Network error: Client connection negotiation failed: client connection to 127.15.250.131:39239: connect: Connection refused (error 111)
W20260808 07:03:20.261667 24813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:20.302019 24813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:20.331148 24813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:20.373456 24813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
W20260808 07:03:20.403352 24813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58114: Illegal state: replica 0383e7089e6448eb84566b41e2415e2e is not leader of this config: current role FOLLOWER
I20260808 07:03:20.410249 25058 raft_consensus.cc:482] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 07:03:20.410372 25058 raft_consensus.cc:504] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "0383e7089e6448eb84566b41e2415e2e" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34401 } } peers { permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 } } peers { permanent_uuid: "dc14e3d81a4e4d44bf00163aaab40610" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 43635 } attrs { } }
I20260808 07:03:20.410676 25058 leader_election.cc:288] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c24e263e9ede48a0bc8c1e84fb8245e7 (127.15.250.131:39239), dc14e3d81a4e4d44bf00163aaab40610 (127.15.250.129:43635)
W20260808 07:03:20.411208 24789 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.250.131:39239: connect: Connection refused (error 111)
W20260808 07:03:20.411764 24789 leader_election.cc:334] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c24e263e9ede48a0bc8c1e84fb8245e7 (127.15.250.131:39239): Network error: Client connection negotiation failed: client connection to 127.15.250.131:39239: connect: Connection refused (error 111)
I20260808 07:03:20.414074 24994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fa711728674c50a616400f2fc53ca1" candidate_uuid: "0383e7089e6448eb84566b41e2415e2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 42 } } ignore_live_leader: false dest_uuid: "dc14e3d81a4e4d44bf00163aaab40610" is_pre_election: true
I20260808 07:03:20.414747 24994 raft_consensus.cc:1789] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 1.1
I20260808 07:03:20.414944 24994 raft_consensus.cc:2454] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0383e7089e6448eb84566b41e2415e2e in term 2.
I20260808 07:03:20.415122 24791 leader_election.cc:302] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0383e7089e6448eb84566b41e2415e2e, dc14e3d81a4e4d44bf00163aaab40610; no voters: c24e263e9ede48a0bc8c1e84fb8245e7
I20260808 07:03:20.415320 25058 raft_consensus.cc:2781] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 07:03:20.415414 25058 raft_consensus.cc:482] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 07:03:20.415452 25058 raft_consensus.cc:3037] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [term 2 FOLLOWER]: Advancing to term 3
I20260808 07:03:20.416695 25058 raft_consensus.cc:504] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "0383e7089e6448eb84566b41e2415e2e" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34401 } } peers { permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 } } peers { permanent_uuid: "dc14e3d81a4e4d44bf00163aaab40610" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 43635 } attrs { } }
I20260808 07:03:20.416872 25058 leader_election.cc:288] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [CANDIDATE]: Term 3 election: Requested vote from peers c24e263e9ede48a0bc8c1e84fb8245e7 (127.15.250.131:39239), dc14e3d81a4e4d44bf00163aaab40610 (127.15.250.129:43635)
I20260808 07:03:20.417038 24994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fa711728674c50a616400f2fc53ca1" candidate_uuid: "0383e7089e6448eb84566b41e2415e2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 42 } } ignore_live_leader: false dest_uuid: "dc14e3d81a4e4d44bf00163aaab40610"
I20260808 07:03:20.417135 24994 raft_consensus.cc:1789] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 1.1
I20260808 07:03:20.417179 24994 raft_consensus.cc:3037] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [term 2 FOLLOWER]: Advancing to term 3
W20260808 07:03:20.417511 24789 leader_election.cc:334] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer c24e263e9ede48a0bc8c1e84fb8245e7 (127.15.250.131:39239): Network error: Client connection negotiation failed: client connection to 127.15.250.131:39239: connect: Connection refused (error 111)
I20260808 07:03:20.417941 24994 raft_consensus.cc:2454] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0383e7089e6448eb84566b41e2415e2e in term 3.
I20260808 07:03:20.418205 24791 leader_election.cc:302] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0383e7089e6448eb84566b41e2415e2e, dc14e3d81a4e4d44bf00163aaab40610; no voters: c24e263e9ede48a0bc8c1e84fb8245e7
I20260808 07:03:20.418323 25058 raft_consensus.cc:2781] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [term 3 FOLLOWER]: Leader election won for term 3
I20260808 07:03:20.418471 25058 raft_consensus.cc:686] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [term 3 LEADER]: Becoming Leader. State: Replica: 0383e7089e6448eb84566b41e2415e2e, State: Running, Role: LEADER
I20260808 07:03:20.418576 25058 consensus_queue.cc:237] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 2.42, Last appended by leader: 42, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "0383e7089e6448eb84566b41e2415e2e" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34401 } } peers { permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 } } peers { permanent_uuid: "dc14e3d81a4e4d44bf00163aaab40610" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 43635 } attrs { } }
I20260808 07:03:20.419431 24709 catalog_manager.cc:5495] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "0383e7089e6448eb84566b41e2415e2e" committed_config { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "0383e7089e6448eb84566b41e2415e2e" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34401 } } peers { permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 } } peers { permanent_uuid: "dc14e3d81a4e4d44bf00163aaab40610" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 43635 } attrs { } } }
W20260808 07:03:20.453539 24791 consensus_peers.cc:487] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e -> Peer dc14e3d81a4e4d44bf00163aaab40610 (127.15.250.129:43635): Couldn't send request to peer dc14e3d81a4e4d44bf00163aaab40610. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260808 07:03:20.454432 24789 consensus_peers.cc:487] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e -> Peer c24e263e9ede48a0bc8c1e84fb8245e7 (127.15.250.131:39239): Couldn't send request to peer c24e263e9ede48a0bc8c1e84fb8245e7. Status: Network error: Client connection negotiation failed: client connection to 127.15.250.131:39239: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 07:03:20.909247 25071 tablet_replica.cc:330] stopping tablet replica
I20260808 07:03:20.909348 25071 raft_consensus.cc:2229] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 07:03:20.909372 25071 raft_consensus.cc:2258] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 07:03:20.909472 25071 ts_tablet_manager.cc:927] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: Initiating tablet copy from peer 0383e7089e6448eb84566b41e2415e2e (127.15.250.130:34401)
I20260808 07:03:20.909725 25071 tablet_copy_client.cc:316] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: tablet copy: Beginning tablet copy session from remote peer at address 127.15.250.130:34401
I20260808 07:03:20.913321 24877 tablet_copy_service.cc:140] P 0383e7089e6448eb84566b41e2415e2e: Received BeginTabletCopySession request for tablet 20fa711728674c50a616400f2fc53ca1 from peer dc14e3d81a4e4d44bf00163aaab40610 ({username='slave'} at 127.15.250.129:42229)
I20260808 07:03:20.913401 24877 tablet_copy_service.cc:161] P 0383e7089e6448eb84566b41e2415e2e: Beginning new tablet copy session on tablet 20fa711728674c50a616400f2fc53ca1 from peer dc14e3d81a4e4d44bf00163aaab40610 at {username='slave'} at 127.15.250.129:42229: session id = dc14e3d81a4e4d44bf00163aaab40610-20fa711728674c50a616400f2fc53ca1
I20260808 07:03:20.914073 24877 tablet_copy_source_session.cc:205] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e: Tablet Copy: opened 5 blocks and 3 log segments
I20260808 07:03:20.914884 25071 ts_tablet_manager.cc:1905] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260808 07:03:20.916467 25071 ts_tablet_manager.cc:1918] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1
I20260808 07:03:20.916606 25071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20fa711728674c50a616400f2fc53ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:20.917264 25071 tablet_copy_client.cc:791] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: tablet copy: Starting download of 5 data blocks...
I20260808 07:03:20.921545 25071 tablet_copy_client.cc:655] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: tablet copy: Starting download of 3 WAL segments...
I20260808 07:03:20.929960 25071 tablet_copy_client.cc:531] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 07:03:20.931007 25071 tablet_bootstrap.cc:492] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: Bootstrap starting.
I20260808 07:03:20.933492 25071 log.cc:824] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:20.936800 25071 tablet_bootstrap.cc:492] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: Bootstrap replayed 1/3 log segments. Stats: ops{read=7 overwritten=0 applied=5 ignored=5} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20260808 07:03:20.938750 25071 tablet_bootstrap.cc:492] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: Bootstrap replayed 2/3 log segments. Stats: ops{read=14 overwritten=0 applied=13 ignored=13} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260808 07:03:20.940083 25071 tablet_bootstrap.cc:492] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: Bootstrap replayed 3/3 log segments. Stats: ops{read=24 overwritten=0 applied=19 ignored=17} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 5 replicates
I20260808 07:03:20.940500 25071 tablet_bootstrap.cc:492] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: Bootstrap complete.
I20260808 07:03:20.941037 25071 ts_tablet_manager.cc:1397] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.001s
I20260808 07:03:20.941551 25071 raft_consensus.cc:348] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [term 3 FOLLOWER]: Replica starting. Triggering 5 pending ops. Active config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "0383e7089e6448eb84566b41e2415e2e" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34401 } } peers { permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 } } peers { permanent_uuid: "dc14e3d81a4e4d44bf00163aaab40610" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 43635 } attrs { } }
I20260808 07:03:20.942094 25071 raft_consensus.cc:729] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc14e3d81a4e4d44bf00163aaab40610, State: Initialized, Role: FOLLOWER
I20260808 07:03:20.942296 25071 consensus_queue.cc:260] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 42, Last appended: 3.47, Last appended by leader: 47, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "0383e7089e6448eb84566b41e2415e2e" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 34401 } } peers { permanent_uuid: "c24e263e9ede48a0bc8c1e84fb8245e7" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 39239 } } peers { permanent_uuid: "dc14e3d81a4e4d44bf00163aaab40610" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 43635 } attrs { } }
I20260808 07:03:20.942603 25071 ts_tablet_manager.cc:1428] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260808 07:03:20.943064 24876 tablet_copy_service.cc:342] P 0383e7089e6448eb84566b41e2415e2e: Request end of tablet copy session dc14e3d81a4e4d44bf00163aaab40610-20fa711728674c50a616400f2fc53ca1 received from {username='slave'} at 127.15.250.129:42229
I20260808 07:03:20.943156 24876 tablet_copy_service.cc:433] P 0383e7089e6448eb84566b41e2415e2e: ending tablet copy session dc14e3d81a4e4d44bf00163aaab40610-20fa711728674c50a616400f2fc53ca1 on tablet 20fa711728674c50a616400f2fc53ca1 with peer dc14e3d81a4e4d44bf00163aaab40610
I20260808 07:03:21.006239 25042 maintenance_manager.cc:419] P dc14e3d81a4e4d44bf00163aaab40610: Scheduling UndoDeltaBlockGCOp(20fa711728674c50a616400f2fc53ca1): 558 bytes on disk
I20260808 07:03:21.006736 24929 maintenance_manager.cc:643] P dc14e3d81a4e4d44bf00163aaab40610: UndoDeltaBlockGCOp(20fa711728674c50a616400f2fc53ca1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":22,"lbm_reads_lt_1ms":4}
I20260808 07:03:21.435128 24994 raft_consensus.cc:1203] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 2.42->[3.43-3.47] Dedup: 3.47->[]
I20260808 07:03:21.608212 25042 maintenance_manager.cc:419] P dc14e3d81a4e4d44bf00163aaab40610: Scheduling LogGCOp(20fa711728674c50a616400f2fc53ca1): free 920390 bytes of WAL
I20260808 07:03:21.608560 24929 log_reader.cc:386] T 20fa711728674c50a616400f2fc53ca1: removed 1 log segments from log reader
I20260808 07:03:21.608631 24929 log.cc:1076] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: Deleting log segment in path: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/wals/20fa711728674c50a616400f2fc53ca1/wal-000000001 (ops 24-30)
I20260808 07:03:21.609025 24929 maintenance_manager.cc:643] P dc14e3d81a4e4d44bf00163aaab40610: LogGCOp(20fa711728674c50a616400f2fc53ca1) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20260808 07:03:21.647745 24906 maintenance_manager.cc:419] P 0383e7089e6448eb84566b41e2415e2e: Scheduling LogGCOp(20fa711728674c50a616400f2fc53ca1): free 920390 bytes of WAL
I20260808 07:03:21.647924 24792 log_reader.cc:386] T 20fa711728674c50a616400f2fc53ca1: removed 1 log segments from log reader
I20260808 07:03:21.648005 24792 log.cc:1076] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e: Deleting log segment in path: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/wals/20fa711728674c50a616400f2fc53ca1/wal-000000001 (ops 24-30)
I20260808 07:03:21.648437 24792 maintenance_manager.cc:643] P 0383e7089e6448eb84566b41e2415e2e: LogGCOp(20fa711728674c50a616400f2fc53ca1) complete. Timing: real 0.001s user 0.000s sys 0.001s Metrics: {}
I20260808 07:03:21.848862 24906 maintenance_manager.cc:419] P 0383e7089e6448eb84566b41e2415e2e: Scheduling LogGCOp(20fa711728674c50a616400f2fc53ca1): free 920510 bytes of WAL
I20260808 07:03:21.849061 24792 log_reader.cc:386] T 20fa711728674c50a616400f2fc53ca1: removed 1 log segments from log reader
I20260808 07:03:21.849097 24792 log.cc:1076] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e: Deleting log segment in path: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/wals/20fa711728674c50a616400f2fc53ca1/wal-000000002 (ops 31-37)
I20260808 07:03:21.849707 24792 maintenance_manager.cc:643] P 0383e7089e6448eb84566b41e2415e2e: LogGCOp(20fa711728674c50a616400f2fc53ca1) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20260808 07:03:21.849855 24906 maintenance_manager.cc:419] P 0383e7089e6448eb84566b41e2415e2e: Scheduling FlushMRSOp(20fa711728674c50a616400f2fc53ca1): 2094765 bytes log retention, and flush 4120343 bytes memory
I20260808 07:03:21.895792 24792 maintenance_manager.cc:643] P 0383e7089e6448eb84566b41e2415e2e: FlushMRSOp(20fa711728674c50a616400f2fc53ca1) complete. Timing: real 0.046s user 0.026s sys 0.014s Metrics: {"bytes_written":881637,"cfile_init":1,"compiler_manager_pool.queue_time_us":177,"dirs.queue_time_us":7550,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2101,"lbm_writes_lt_1ms":57,"peak_mem_usage":0,"rows_written":31000,"thread_start_us":230,"threads_started":2}
I20260808 07:03:21.896071 24906 maintenance_manager.cc:419] P 0383e7089e6448eb84566b41e2415e2e: Scheduling CompactRowSetsOp(20fa711728674c50a616400f2fc53ca1): perf score=0.949984
I20260808 07:03:21.909530 25042 maintenance_manager.cc:419] P dc14e3d81a4e4d44bf00163aaab40610: Scheduling LogGCOp(20fa711728674c50a616400f2fc53ca1): free 920510 bytes of WAL
I20260808 07:03:21.909693 24929 log_reader.cc:386] T 20fa711728674c50a616400f2fc53ca1: removed 1 log segments from log reader
I20260808 07:03:21.909754 24929 log.cc:1076] T 20fa711728674c50a616400f2fc53ca1 P dc14e3d81a4e4d44bf00163aaab40610: Deleting log segment in path: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/wals/20fa711728674c50a616400f2fc53ca1/wal-000000002 (ops 31-37)
I20260808 07:03:21.910138 24929 maintenance_manager.cc:643] P dc14e3d81a4e4d44bf00163aaab40610: LogGCOp(20fa711728674c50a616400f2fc53ca1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260808 07:03:21.910251 25042 maintenance_manager.cc:419] P dc14e3d81a4e4d44bf00163aaab40610: Scheduling FlushMRSOp(20fa711728674c50a616400f2fc53ca1): 2091949 bytes log retention, and flush 4120343 bytes memory
I20260808 07:03:21.939534 24792 maintenance_manager.cc:643] P 0383e7089e6448eb84566b41e2415e2e: CompactRowSetsOp(20fa711728674c50a616400f2fc53ca1) complete. Timing: real 0.043s user 0.031s sys 0.007s Metrics: {"bytes_written":4680169,"cfile_cache_hit":11,"cfile_cache_hit_bytes":484040,"cfile_cache_miss":57,"cfile_cache_miss_bytes":1995586,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":578,"dirs.run_cpu_time_us":153,"dirs.run_wall_time_us":1415,"drs_written":1,"lbm_read_time_us":1233,"lbm_reads_lt_1ms":81,"lbm_write_time_us":7475,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":69,"num_input_rowsets":2,"peak_mem_usage":470147,"rows_written":31038,"thread_start_us":296,"threads_started":5}
I20260808 07:03:21.940308 24906 maintenance_manager.cc:419] P 0383e7089e6448eb84566b41e2415e2e: Scheduling UndoDeltaBlockGCOp(20fa711728674c50a616400f2fc53ca1): 305303 bytes on disk
I20260808 07:03:21.940603 24792 maintenance_manager.cc:643] P 0383e7089e6448eb84566b41e2415e2e: UndoDeltaBlockGCOp(20fa711728674c50a616400f2fc53ca1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20260808 07:03:22.002736 24929 maintenance_manager.cc:643] P dc14e3d81a4e4d44bf00163aaab40610: FlushMRSOp(20fa711728674c50a616400f2fc53ca1) complete. Timing: real 0.092s user 0.036s sys 0.011s Metrics: {"bytes_written":947295,"cfile_init":1,"compiler_manager_pool.queue_time_us":12032,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3006,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":58,"peak_mem_usage":0,"rows_written":33400,"thread_start_us":141,"threads_started":2}
I20260808 07:03:22.003100 25042 maintenance_manager.cc:419] P dc14e3d81a4e4d44bf00163aaab40610: Scheduling UndoDeltaBlockGCOp(20fa711728674c50a616400f2fc53ca1): 330841 bytes on disk
I20260808 07:03:22.003309 24929 maintenance_manager.cc:643] P dc14e3d81a4e4d44bf00163aaab40610: UndoDeltaBlockGCOp(20fa711728674c50a616400f2fc53ca1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":25,"lbm_reads_lt_1ms":4}
I20260808 07:03:22.003481 25042 maintenance_manager.cc:419] P dc14e3d81a4e4d44bf00163aaab40610: Scheduling CompactRowSetsOp(20fa711728674c50a616400f2fc53ca1): perf score=0.949305
I20260808 07:03:22.050158 25063 consensus_queue.cc:786] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e [LEADER]: Peer c24e263e9ede48a0bc8c1e84fb8245e7 is lagging by at least 365 ops behind the committed index
I20260808 07:03:22.055303 24929 maintenance_manager.cc:643] P dc14e3d81a4e4d44bf00163aaab40610: CompactRowSetsOp(20fa711728674c50a616400f2fc53ca1) complete. Timing: real 0.051s user 0.030s sys 0.009s Metrics: {"bytes_written":4745477,"cfile_cache_hit":11,"cfile_cache_hit_bytes":490556,"cfile_cache_miss":59,"cfile_cache_miss_bytes":2164282,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":67,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":792,"drs_written":1,"lbm_read_time_us":1205,"lbm_reads_lt_1ms":79,"lbm_write_time_us":7921,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":71,"mutex_wait_us":49,"num_input_rowsets":2,"peak_mem_usage":506740,"rows_written":33438,"thread_start_us":253,"threads_started":4}
I20260808 07:03:22.055668 25042 maintenance_manager.cc:419] P dc14e3d81a4e4d44bf00163aaab40610: Scheduling UndoDeltaBlockGCOp(20fa711728674c50a616400f2fc53ca1): 331316 bytes on disk
I20260808 07:03:22.055871 24929 maintenance_manager.cc:643] P dc14e3d81a4e4d44bf00163aaab40610: UndoDeltaBlockGCOp(20fa711728674c50a616400f2fc53ca1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4}
I20260808 07:03:22.241420 24906 maintenance_manager.cc:419] P 0383e7089e6448eb84566b41e2415e2e: Scheduling LogGCOp(20fa711728674c50a616400f2fc53ca1): free 1048418 bytes of WAL
I20260808 07:03:22.241591 24792 log_reader.cc:386] T 20fa711728674c50a616400f2fc53ca1: removed 1 log segments from log reader
I20260808 07:03:22.241652 24792 log.cc:1076] T 20fa711728674c50a616400f2fc53ca1 P 0383e7089e6448eb84566b41e2415e2e: Deleting log segment in path: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/wals/20fa711728674c50a616400f2fc53ca1/wal-000000003 (ops 38-275)
I20260808 07:03:22.242120 24792 maintenance_manager.cc:643] P 0383e7089e6448eb84566b41e2415e2e: LogGCOp(20fa711728674c50a616400f2fc53ca1) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20260808 07:03:22.244962 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 24909
I20260808 07:03:22.252992 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 24749
I20260808 07:03:22.260733 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 24678
2026-08-08T07:03:22Z chronyd exiting
[ OK ] RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy (9699 ms)
[ RUN ] RaftConsensusElectionITest.DisruptiveServerAndSlowWAL
I20260808 07:03:22.286708 16362 test_util.cc:268] Using random seed: -1216904593
I20260808 07:03:22.287295 16362 ts_itest-base.cc:112] Starting cluster with:
I20260808 07:03:22.287348 16362 ts_itest-base.cc:113] --------------
I20260808 07:03:22.287367 16362 ts_itest-base.cc:114] 3 tablet servers
I20260808 07:03:22.287393 16362 ts_itest-base.cc:115] 3 replicas per TS
I20260808 07:03:22.287418 16362 ts_itest-base.cc:116] --------------
2026-08-08T07:03:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:03:22Z Disabled control of system clock
I20260808 07:03:22.297132 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.250.190:42427
--webserver_interface=127.15.250.190
--webserver_port=0
--builtin_ntp_servers=127.15.250.148:36907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.250.190:42427 with env {}
W20260808 07:03:22.380903 25124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:22.381170 25124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:22.381206 25124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:22.382406 25124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:03:22.382480 25124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:22.382503 25124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:03:22.382524 25124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:03:22.383946 25124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:36907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.250.190:42427
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.250.190:42427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.250.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.25124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:22.384244 25124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:22.384660 25124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:22.389931 25129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:22.389946 25132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:22.389946 25130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:22.390188 25124 server_base.cc:1034] running on GCE node
I20260808 07:03:22.390435 25124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:22.390753 25124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:22.391901 25124 hybrid_clock.cc:648] HybridClock initialized: now 1786172602391874 us; error 38 us; skew 500 ppm
I20260808 07:03:22.393342 25124 webserver.cc:460] Webserver started at http://127.15.250.190:43035/ using document root <none> and password file <none>
I20260808 07:03:22.393618 25124 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:22.393688 25124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:22.393874 25124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:22.394907 25124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ac96500bc99c4f26b56cd4e6ad7ab3e3"
format_stamp: "Formatted at 2026-08-08 07:03:22 on dist-test-slave-0b58"
I20260808 07:03:22.395293 25124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ac96500bc99c4f26b56cd4e6ad7ab3e3"
format_stamp: "Formatted at 2026-08-08 07:03:22 on dist-test-slave-0b58"
I20260808 07:03:22.397105 25124 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260808 07:03:22.398031 25138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:22.398242 25124 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:03:22.398378 25124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ac96500bc99c4f26b56cd4e6ad7ab3e3"
format_stamp: "Formatted at 2026-08-08 07:03:22 on dist-test-slave-0b58"
I20260808 07:03:22.398437 25124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:22.419821 25124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:22.420161 25124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:22.420378 25124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:22.424449 25124 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.190:42427
I20260808 07:03:22.424517 25191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.190:42427 every 8 connection(s)
I20260808 07:03:22.424855 25124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260808 07:03:22.425635 25192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:22.428421 25192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac96500bc99c4f26b56cd4e6ad7ab3e3: Bootstrap starting.
I20260808 07:03:22.429073 25192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac96500bc99c4f26b56cd4e6ad7ab3e3: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:22.429394 25192 log.cc:824] T 00000000000000000000000000000000 P ac96500bc99c4f26b56cd4e6ad7ab3e3: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:22.430096 25192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac96500bc99c4f26b56cd4e6ad7ab3e3: No bootstrap required, opened a new log
I20260808 07:03:22.431861 25192 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac96500bc99c4f26b56cd4e6ad7ab3e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac96500bc99c4f26b56cd4e6ad7ab3e3" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 42427 } }
I20260808 07:03:22.432008 25192 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac96500bc99c4f26b56cd4e6ad7ab3e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:03:22.432031 25192 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac96500bc99c4f26b56cd4e6ad7ab3e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac96500bc99c4f26b56cd4e6ad7ab3e3, State: Initialized, Role: FOLLOWER
I20260808 07:03:22.432210 25192 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac96500bc99c4f26b56cd4e6ad7ab3e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac96500bc99c4f26b56cd4e6ad7ab3e3" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 42427 } }
I20260808 07:03:22.432351 25192 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac96500bc99c4f26b56cd4e6ad7ab3e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:03:22.432406 25192 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac96500bc99c4f26b56cd4e6ad7ab3e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:03:22.432463 25192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac96500bc99c4f26b56cd4e6ad7ab3e3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:22.432518 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 25124
I20260808 07:03:22.432638 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260808 07:03:22.433121 25192 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac96500bc99c4f26b56cd4e6ad7ab3e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac96500bc99c4f26b56cd4e6ad7ab3e3" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 42427 } }
I20260808 07:03:22.433262 25192 leader_election.cc:302] T 00000000000000000000000000000000 P ac96500bc99c4f26b56cd4e6ad7ab3e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac96500bc99c4f26b56cd4e6ad7ab3e3; no voters:
I20260808 07:03:22.433696 25192 leader_election.cc:288] T 00000000000000000000000000000000 P ac96500bc99c4f26b56cd4e6ad7ab3e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:03:22.433763 25198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac96500bc99c4f26b56cd4e6ad7ab3e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:22.433949 25198 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac96500bc99c4f26b56cd4e6ad7ab3e3 [term 1 LEADER]: Becoming Leader. State: Replica: ac96500bc99c4f26b56cd4e6ad7ab3e3, State: Running, Role: LEADER
I20260808 07:03:22.433952 25192 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac96500bc99c4f26b56cd4e6ad7ab3e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:03:22.434083 25198 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac96500bc99c4f26b56cd4e6ad7ab3e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac96500bc99c4f26b56cd4e6ad7ab3e3" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 42427 } }
I20260808 07:03:22.434484 25200 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac96500bc99c4f26b56cd4e6ad7ab3e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac96500bc99c4f26b56cd4e6ad7ab3e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac96500bc99c4f26b56cd4e6ad7ab3e3" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 42427 } } }
I20260808 07:03:22.434476 25198 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac96500bc99c4f26b56cd4e6ad7ab3e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac96500bc99c4f26b56cd4e6ad7ab3e3. Latest consensus state: current_term: 1 leader_uuid: "ac96500bc99c4f26b56cd4e6ad7ab3e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac96500bc99c4f26b56cd4e6ad7ab3e3" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 42427 } } }
I20260808 07:03:22.434568 25200 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac96500bc99c4f26b56cd4e6ad7ab3e3 [sys.catalog]: This master's current role is: LEADER
I20260808 07:03:22.434576 25198 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac96500bc99c4f26b56cd4e6ad7ab3e3 [sys.catalog]: This master's current role is: LEADER
I20260808 07:03:22.435302 25206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:03:22.436002 25206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:03:22.437680 25206 catalog_manager.cc:1347] Generated new cluster ID: 31d2b0031fa047b08b3b7dd6b19eb170
I20260808 07:03:22.437738 25206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:03:22.463176 25206 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:03:22.463949 25206 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:03:22.467048 25206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac96500bc99c4f26b56cd4e6ad7ab3e3: Generated new TSK 0
I20260808 07:03:22.467237 25206 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:03:22.472651 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.129:0
--local_ip_for_outbound_sockets=127.15.250.129
--webserver_interface=127.15.250.129
--webserver_port=0
--tserver_master_addrs=127.15.250.190:42427
--builtin_ntp_servers=127.15.250.148:36907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--leader_failure_max_missed_heartbeat_periods=3 with env {}
W20260808 07:03:22.557309 25217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:22.557561 25217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:22.557595 25217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:22.558751 25217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:22.558843 25217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.129
I20260808 07:03:22.560168 25217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:36907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.250.129
--webserver_port=0
--tserver_master_addrs=127.15.250.190:42427
--never_fsync=true
--heap_profile_path=/tmp/kudu.25217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.129
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:22.560518 25217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:22.560840 25217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:22.566107 25222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:22.566097 25225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:22.566253 25223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:22.566150 25217 server_base.cc:1034] running on GCE node
I20260808 07:03:22.566592 25217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:22.566849 25217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:22.568020 25217 hybrid_clock.cc:648] HybridClock initialized: now 1786172602568003 us; error 36 us; skew 500 ppm
I20260808 07:03:22.569578 25217 webserver.cc:460] Webserver started at http://127.15.250.129:45397/ using document root <none> and password file <none>
I20260808 07:03:22.569840 25217 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:22.569898 25217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:22.570021 25217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:22.571287 25217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "7b73901566a14a56b74acf52ff5d232e"
format_stamp: "Formatted at 2026-08-08 07:03:22 on dist-test-slave-0b58"
I20260808 07:03:22.571731 25217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "7b73901566a14a56b74acf52ff5d232e"
format_stamp: "Formatted at 2026-08-08 07:03:22 on dist-test-slave-0b58"
I20260808 07:03:22.573294 25217 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 07:03:22.574060 25231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:22.574283 25217 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 07:03:22.574376 25217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "7b73901566a14a56b74acf52ff5d232e"
format_stamp: "Formatted at 2026-08-08 07:03:22 on dist-test-slave-0b58"
I20260808 07:03:22.574437 25217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:22.594839 25217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:22.595134 25217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:22.595248 25217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:22.595446 25217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:03:22.595803 25217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:03:22.595839 25217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:22.595898 25217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:03:22.595924 25217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:22.602438 25217 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.129:45813
I20260808 07:03:22.602475 25348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.129:45813 every 8 connection(s)
I20260808 07:03:22.602857 25217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260808 07:03:22.608942 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 25217
I20260808 07:03:22.609066 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260808 07:03:22.609323 25349 heartbeater.cc:347] Connected to a master server at 127.15.250.190:42427
I20260808 07:03:22.609444 25349 heartbeater.cc:464] Registering TS with master...
I20260808 07:03:22.609697 25349 heartbeater.cc:511] Master 127.15.250.190:42427 requested a full tablet report, sending...
I20260808 07:03:22.610215 25155 ts_manager.cc:194] Registered new tserver with Master: 7b73901566a14a56b74acf52ff5d232e (127.15.250.129:45813)
I20260808 07:03:22.610390 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.130:0
--local_ip_for_outbound_sockets=127.15.250.130
--webserver_interface=127.15.250.130
--webserver_port=0
--tserver_master_addrs=127.15.250.190:42427
--builtin_ntp_servers=127.15.250.148:36907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--leader_failure_max_missed_heartbeat_periods=3 with env {}
I20260808 07:03:22.610934 25155 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.129:37065
W20260808 07:03:22.695142 25352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:22.695381 25352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:22.695402 25352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:22.696686 25352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:22.696744 25352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.130
I20260808 07:03:22.698124 25352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:36907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.250.130
--webserver_port=0
--tserver_master_addrs=127.15.250.190:42427
--never_fsync=true
--heap_profile_path=/tmp/kudu.25352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.130
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:22.698403 25352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:22.698732 25352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:22.704425 25360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:22.704391 25357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:22.704407 25358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:22.704552 25352 server_base.cc:1034] running on GCE node
I20260808 07:03:22.704907 25352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:22.705127 25352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:22.706308 25352 hybrid_clock.cc:648] HybridClock initialized: now 1786172602706298 us; error 34 us; skew 500 ppm
I20260808 07:03:22.707732 25352 webserver.cc:460] Webserver started at http://127.15.250.130:39519/ using document root <none> and password file <none>
I20260808 07:03:22.707962 25352 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:22.708006 25352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:22.708108 25352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:22.709190 25352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "f112c659629f409a812d76076989e924"
format_stamp: "Formatted at 2026-08-08 07:03:22 on dist-test-slave-0b58"
I20260808 07:03:22.709578 25352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "f112c659629f409a812d76076989e924"
format_stamp: "Formatted at 2026-08-08 07:03:22 on dist-test-slave-0b58"
I20260808 07:03:22.710902 25352 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 07:03:22.711647 25366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:22.711849 25352 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 07:03:22.711956 25352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "f112c659629f409a812d76076989e924"
format_stamp: "Formatted at 2026-08-08 07:03:22 on dist-test-slave-0b58"
I20260808 07:03:22.712049 25352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:22.747931 25352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:22.748377 25352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:22.748548 25352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:22.748817 25352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:03:22.749200 25352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:03:22.749258 25352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:22.749295 25352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:03:22.749322 25352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:22.755206 25352 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.130:38349
I20260808 07:03:22.755250 25481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.130:38349 every 8 connection(s)
I20260808 07:03:22.755641 25352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260808 07:03:22.756865 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 25352
I20260808 07:03:22.756975 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260808 07:03:22.758342 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.131:0
--local_ip_for_outbound_sockets=127.15.250.131
--webserver_interface=127.15.250.131
--webserver_port=0
--tserver_master_addrs=127.15.250.190:42427
--builtin_ntp_servers=127.15.250.148:36907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--leader_failure_max_missed_heartbeat_periods=3 with env {}
I20260808 07:03:22.762687 25482 heartbeater.cc:347] Connected to a master server at 127.15.250.190:42427
I20260808 07:03:22.762780 25482 heartbeater.cc:464] Registering TS with master...
I20260808 07:03:22.762964 25482 heartbeater.cc:511] Master 127.15.250.190:42427 requested a full tablet report, sending...
I20260808 07:03:22.763444 25155 ts_manager.cc:194] Registered new tserver with Master: f112c659629f409a812d76076989e924 (127.15.250.130:38349)
I20260808 07:03:22.763846 25155 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.130:40815
W20260808 07:03:22.843266 25485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:22.843550 25485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:22.843590 25485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:22.844805 25485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:22.844893 25485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.131
I20260808 07:03:22.846290 25485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:36907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.250.131
--webserver_port=0
--tserver_master_addrs=127.15.250.190:42427
--never_fsync=true
--heap_profile_path=/tmp/kudu.25485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.131
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:22.846577 25485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:22.846992 25485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:22.852494 25491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:22.852514 25492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:22.852510 25494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:22.852553 25485 server_base.cc:1034] running on GCE node
I20260808 07:03:22.852927 25485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:22.853125 25485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:22.854254 25485 hybrid_clock.cc:648] HybridClock initialized: now 1786172602854222 us; error 44 us; skew 500 ppm
I20260808 07:03:22.855556 25485 webserver.cc:460] Webserver started at http://127.15.250.131:34799/ using document root <none> and password file <none>
I20260808 07:03:22.855823 25485 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:22.855873 25485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:22.856026 25485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:22.857082 25485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "78a3294ee273426aa470db2adbf7f2e1"
format_stamp: "Formatted at 2026-08-08 07:03:22 on dist-test-slave-0b58"
I20260808 07:03:22.857460 25485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "78a3294ee273426aa470db2adbf7f2e1"
format_stamp: "Formatted at 2026-08-08 07:03:22 on dist-test-slave-0b58"
I20260808 07:03:22.858829 25485 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 07:03:22.859617 25500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:22.859809 25485 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:03:22.859912 25485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "78a3294ee273426aa470db2adbf7f2e1"
format_stamp: "Formatted at 2026-08-08 07:03:22 on dist-test-slave-0b58"
I20260808 07:03:22.859997 25485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:22.881915 25485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:22.882288 25485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:22.882460 25485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:22.882714 25485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:03:22.883085 25485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:03:22.883148 25485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:22.883185 25485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:03:22.883234 25485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:22.889430 25485 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.131:45709
I20260808 07:03:22.889473 25615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.131:45709 every 8 connection(s)
I20260808 07:03:22.889837 25485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260808 07:03:22.895344 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 25485
I20260808 07:03:22.895498 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260808 07:03:22.895686 25616 heartbeater.cc:347] Connected to a master server at 127.15.250.190:42427
I20260808 07:03:22.895807 25616 heartbeater.cc:464] Registering TS with master...
I20260808 07:03:22.896003 25616 heartbeater.cc:511] Master 127.15.250.190:42427 requested a full tablet report, sending...
I20260808 07:03:22.896456 25155 ts_manager.cc:194] Registered new tserver with Master: 78a3294ee273426aa470db2adbf7f2e1 (127.15.250.131:45709)
I20260808 07:03:22.896915 25155 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.131:53425
I20260808 07:03:22.896966 16362 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 07:03:22.903401 25155 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35270:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20260808 07:03:22.905203 25155 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 07:03:22.912864 25548 tablet_service.cc:1467] Processing CreateTablet for tablet 3b987c80e5dd489382b90f8d23d6975f (DEFAULT_TABLE table=TestTable [id=d580154ec04343c6914da71264c9f3ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:03:22.913160 25281 tablet_service.cc:1467] Processing CreateTablet for tablet 3b987c80e5dd489382b90f8d23d6975f (DEFAULT_TABLE table=TestTable [id=d580154ec04343c6914da71264c9f3ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:03:22.913297 25548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b987c80e5dd489382b90f8d23d6975f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:22.913502 25281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b987c80e5dd489382b90f8d23d6975f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:22.913570 25415 tablet_service.cc:1467] Processing CreateTablet for tablet 3b987c80e5dd489382b90f8d23d6975f (DEFAULT_TABLE table=TestTable [id=d580154ec04343c6914da71264c9f3ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:03:22.913925 25415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b987c80e5dd489382b90f8d23d6975f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:22.916874 25634 tablet_bootstrap.cc:492] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1: Bootstrap starting.
I20260808 07:03:22.917099 25633 tablet_bootstrap.cc:492] T 3b987c80e5dd489382b90f8d23d6975f P f112c659629f409a812d76076989e924: Bootstrap starting.
I20260808 07:03:22.917702 25633 tablet_bootstrap.cc:654] T 3b987c80e5dd489382b90f8d23d6975f P f112c659629f409a812d76076989e924: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:22.917703 25634 tablet_bootstrap.cc:654] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:22.918005 25634 log.cc:824] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:22.918011 25633 log.cc:824] T 3b987c80e5dd489382b90f8d23d6975f P f112c659629f409a812d76076989e924: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:22.918399 25631 tablet_bootstrap.cc:492] T 3b987c80e5dd489382b90f8d23d6975f P 7b73901566a14a56b74acf52ff5d232e: Bootstrap starting.
I20260808 07:03:22.918699 25634 tablet_bootstrap.cc:492] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1: No bootstrap required, opened a new log
I20260808 07:03:22.918726 25633 tablet_bootstrap.cc:492] T 3b987c80e5dd489382b90f8d23d6975f P f112c659629f409a812d76076989e924: No bootstrap required, opened a new log
I20260808 07:03:22.918788 25634 ts_tablet_manager.cc:1397] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:03:22.918867 25633 ts_tablet_manager.cc:1397] T 3b987c80e5dd489382b90f8d23d6975f P f112c659629f409a812d76076989e924: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 07:03:22.919063 25631 tablet_bootstrap.cc:654] T 3b987c80e5dd489382b90f8d23d6975f P 7b73901566a14a56b74acf52ff5d232e: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:22.919407 25631 log.cc:824] T 3b987c80e5dd489382b90f8d23d6975f P 7b73901566a14a56b74acf52ff5d232e: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:22.920224 25631 tablet_bootstrap.cc:492] T 3b987c80e5dd489382b90f8d23d6975f P 7b73901566a14a56b74acf52ff5d232e: No bootstrap required, opened a new log
I20260808 07:03:22.920346 25631 ts_tablet_manager.cc:1397] T 3b987c80e5dd489382b90f8d23d6975f P 7b73901566a14a56b74acf52ff5d232e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:03:22.921094 25634 raft_consensus.cc:348] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a3294ee273426aa470db2adbf7f2e1" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 45709 } } peers { permanent_uuid: "7b73901566a14a56b74acf52ff5d232e" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45813 } } peers { permanent_uuid: "f112c659629f409a812d76076989e924" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 38349 } }
I20260808 07:03:22.921253 25634 raft_consensus.cc:374] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:03:22.921316 25634 raft_consensus.cc:729] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78a3294ee273426aa470db2adbf7f2e1, State: Initialized, Role: FOLLOWER
I20260808 07:03:22.921464 25634 consensus_queue.cc:260] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a3294ee273426aa470db2adbf7f2e1" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 45709 } } peers { permanent_uuid: "7b73901566a14a56b74acf52ff5d232e" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45813 } } peers { permanent_uuid: "f112c659629f409a812d76076989e924" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 38349 } }
I20260808 07:03:22.921753 25634 ts_tablet_manager.cc:1428] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 07:03:22.921629 25633 raft_consensus.cc:348] T 3b987c80e5dd489382b90f8d23d6975f P f112c659629f409a812d76076989e924 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a3294ee273426aa470db2adbf7f2e1" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 45709 } } peers { permanent_uuid: "7b73901566a14a56b74acf52ff5d232e" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45813 } } peers { permanent_uuid: "f112c659629f409a812d76076989e924" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 38349 } }
I20260808 07:03:22.921813 25633 raft_consensus.cc:374] T 3b987c80e5dd489382b90f8d23d6975f P f112c659629f409a812d76076989e924 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:03:22.921891 25616 heartbeater.cc:503] Master 127.15.250.190:42427 was elected leader, sending a full tablet report...
I20260808 07:03:22.921898 25633 raft_consensus.cc:729] T 3b987c80e5dd489382b90f8d23d6975f P f112c659629f409a812d76076989e924 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f112c659629f409a812d76076989e924, State: Initialized, Role: FOLLOWER
I20260808 07:03:22.922015 25633 consensus_queue.cc:260] T 3b987c80e5dd489382b90f8d23d6975f P f112c659629f409a812d76076989e924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a3294ee273426aa470db2adbf7f2e1" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 45709 } } peers { permanent_uuid: "7b73901566a14a56b74acf52ff5d232e" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45813 } } peers { permanent_uuid: "f112c659629f409a812d76076989e924" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 38349 } }
I20260808 07:03:22.922272 25633 ts_tablet_manager.cc:1428] T 3b987c80e5dd489382b90f8d23d6975f P f112c659629f409a812d76076989e924: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 07:03:22.922317 25482 heartbeater.cc:503] Master 127.15.250.190:42427 was elected leader, sending a full tablet report...
I20260808 07:03:22.922935 25631 raft_consensus.cc:348] T 3b987c80e5dd489382b90f8d23d6975f P 7b73901566a14a56b74acf52ff5d232e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a3294ee273426aa470db2adbf7f2e1" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 45709 } } peers { permanent_uuid: "7b73901566a14a56b74acf52ff5d232e" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45813 } } peers { permanent_uuid: "f112c659629f409a812d76076989e924" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 38349 } }
I20260808 07:03:22.923159 25631 raft_consensus.cc:374] T 3b987c80e5dd489382b90f8d23d6975f P 7b73901566a14a56b74acf52ff5d232e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:03:22.923221 25631 raft_consensus.cc:729] T 3b987c80e5dd489382b90f8d23d6975f P 7b73901566a14a56b74acf52ff5d232e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b73901566a14a56b74acf52ff5d232e, State: Initialized, Role: FOLLOWER
I20260808 07:03:22.923352 25631 consensus_queue.cc:260] T 3b987c80e5dd489382b90f8d23d6975f P 7b73901566a14a56b74acf52ff5d232e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a3294ee273426aa470db2adbf7f2e1" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 45709 } } peers { permanent_uuid: "7b73901566a14a56b74acf52ff5d232e" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45813 } } peers { permanent_uuid: "f112c659629f409a812d76076989e924" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 38349 } }
I20260808 07:03:22.923641 25631 ts_tablet_manager.cc:1428] T 3b987c80e5dd489382b90f8d23d6975f P 7b73901566a14a56b74acf52ff5d232e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 07:03:22.923702 25349 heartbeater.cc:503] Master 127.15.250.190:42427 was elected leader, sending a full tablet report...
W20260808 07:03:23.006278 25483 tablet.cc:2369] T 3b987c80e5dd489382b90f8d23d6975f P f112c659629f409a812d76076989e924: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 07:03:23.029004 25638 raft_consensus.cc:482] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 07:03:23.029183 25638 raft_consensus.cc:504] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a3294ee273426aa470db2adbf7f2e1" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 45709 } } peers { permanent_uuid: "7b73901566a14a56b74acf52ff5d232e" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45813 } } peers { permanent_uuid: "f112c659629f409a812d76076989e924" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 38349 } }
I20260808 07:03:23.029556 25638 leader_election.cc:288] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b73901566a14a56b74acf52ff5d232e (127.15.250.129:45813), f112c659629f409a812d76076989e924 (127.15.250.130:38349)
I20260808 07:03:23.033115 25436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "78a3294ee273426aa470db2adbf7f2e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f112c659629f409a812d76076989e924" is_pre_election: true
I20260808 07:03:23.033083 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "78a3294ee273426aa470db2adbf7f2e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: true
I20260808 07:03:23.033272 25302 raft_consensus.cc:2454] T 3b987c80e5dd489382b90f8d23d6975f P 7b73901566a14a56b74acf52ff5d232e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78a3294ee273426aa470db2adbf7f2e1 in term 0.
I20260808 07:03:23.033295 25436 raft_consensus.cc:2454] T 3b987c80e5dd489382b90f8d23d6975f P f112c659629f409a812d76076989e924 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78a3294ee273426aa470db2adbf7f2e1 in term 0.
I20260808 07:03:23.033521 25501 leader_election.cc:302] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78a3294ee273426aa470db2adbf7f2e1, f112c659629f409a812d76076989e924; no voters:
I20260808 07:03:23.033650 25638 raft_consensus.cc:2781] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 07:03:23.033752 25638 raft_consensus.cc:482] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 07:03:23.033789 25638 raft_consensus.cc:3037] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:23.034672 25638 raft_consensus.cc:504] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a3294ee273426aa470db2adbf7f2e1" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 45709 } } peers { permanent_uuid: "7b73901566a14a56b74acf52ff5d232e" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45813 } } peers { permanent_uuid: "f112c659629f409a812d76076989e924" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 38349 } }
I20260808 07:03:23.034816 25638 leader_election.cc:288] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1 [CANDIDATE]: Term 1 election: Requested vote from peers 7b73901566a14a56b74acf52ff5d232e (127.15.250.129:45813), f112c659629f409a812d76076989e924 (127.15.250.130:38349)
I20260808 07:03:23.034986 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "78a3294ee273426aa470db2adbf7f2e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e"
I20260808 07:03:23.035068 25436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "78a3294ee273426aa470db2adbf7f2e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f112c659629f409a812d76076989e924"
I20260808 07:03:23.035082 25302 raft_consensus.cc:3037] T 3b987c80e5dd489382b90f8d23d6975f P 7b73901566a14a56b74acf52ff5d232e [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:23.035157 25436 raft_consensus.cc:3037] T 3b987c80e5dd489382b90f8d23d6975f P f112c659629f409a812d76076989e924 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:23.035902 25302 raft_consensus.cc:2454] T 3b987c80e5dd489382b90f8d23d6975f P 7b73901566a14a56b74acf52ff5d232e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78a3294ee273426aa470db2adbf7f2e1 in term 1.
I20260808 07:03:23.035959 25436 raft_consensus.cc:2454] T 3b987c80e5dd489382b90f8d23d6975f P f112c659629f409a812d76076989e924 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78a3294ee273426aa470db2adbf7f2e1 in term 1.
I20260808 07:03:23.036134 25501 leader_election.cc:302] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78a3294ee273426aa470db2adbf7f2e1, f112c659629f409a812d76076989e924; no voters:
I20260808 07:03:23.036218 25638 raft_consensus.cc:2781] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:23.036391 25638 raft_consensus.cc:686] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1 [term 1 LEADER]: Becoming Leader. State: Replica: 78a3294ee273426aa470db2adbf7f2e1, State: Running, Role: LEADER
I20260808 07:03:23.036504 25638 consensus_queue.cc:237] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a3294ee273426aa470db2adbf7f2e1" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 45709 } } peers { permanent_uuid: "7b73901566a14a56b74acf52ff5d232e" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45813 } } peers { permanent_uuid: "f112c659629f409a812d76076989e924" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 38349 } }
I20260808 07:03:23.037328 25155 catalog_manager.cc:5495] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78a3294ee273426aa470db2adbf7f2e1 (127.15.250.131). New cstate: current_term: 1 leader_uuid: "78a3294ee273426aa470db2adbf7f2e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a3294ee273426aa470db2adbf7f2e1" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 45709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b73901566a14a56b74acf52ff5d232e" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f112c659629f409a812d76076989e924" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 38349 } health_report { overall_health: UNKNOWN } } }
I20260808 07:03:23.059078 16362 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 07:03:23.059895 16362 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 7b73901566a14a56b74acf52ff5d232e to finish bootstrapping
I20260808 07:03:23.064100 16362 ts_itest-base.cc:233] Waiting for 1 tablets on tserver f112c659629f409a812d76076989e924 to finish bootstrapping
I20260808 07:03:23.068094 16362 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 78a3294ee273426aa470db2adbf7f2e1 to finish bootstrapping
I20260808 07:03:23.072588 25343 generic_service.cc:175] {username='slave'} at 127.0.0.1:58816 changed flags via RPC: log_inject_latency from 'false' to 'false'
I20260808 07:03:23.073192 25476 generic_service.cc:175] {username='slave'} at 127.0.0.1:55832 changed flags via RPC: log_inject_latency from 'false' to 'false'
I20260808 07:03:23.073764 25610 generic_service.cc:175] {username='slave'} at 127.0.0.1:40398 changed flags via RPC: log_inject_latency from 'false' to 'false'
I20260808 07:03:23.080032 25436 raft_consensus.cc:1261] T 3b987c80e5dd489382b90f8d23d6975f P f112c659629f409a812d76076989e924 [term 1 FOLLOWER]: Refusing update from remote peer 78a3294ee273426aa470db2adbf7f2e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 07:03:23.080054 25302 raft_consensus.cc:1261] T 3b987c80e5dd489382b90f8d23d6975f P 7b73901566a14a56b74acf52ff5d232e [term 1 FOLLOWER]: Refusing update from remote peer 78a3294ee273426aa470db2adbf7f2e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 07:03:23.080565 25638 consensus_queue.cc:1035] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f112c659629f409a812d76076989e924" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:03:23.080736 25642 consensus_queue.cc:1035] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b73901566a14a56b74acf52ff5d232e" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:03:23.084048 25648 mvcc.cc:205] Tried to move back new op lower bound from 7316162982213136384 to 7316162982038114304. Current Snapshot: MvccSnapshot[applied={T|T < 7316162982213136384 or (T in {7316162982213136384})}]
I20260808 07:03:23.091446 25610 generic_service.cc:175] {username='slave'} at 127.0.0.1:40398 changed flags via RPC: log_inject_latency_ms_mean from '100' to '1200'
I20260808 07:03:23.091915 25610 generic_service.cc:175] {username='slave'} at 127.0.0.1:40398 changed flags via RPC: log_inject_latency_ms_stddev from '100' to '0'
I20260808 07:03:23.092342 25610 generic_service.cc:175] {username='slave'} at 127.0.0.1:40398 changed flags via RPC: log_inject_latency from 'false' to 'true'
I20260808 07:03:23.092767 25343 generic_service.cc:175] {username='slave'} at 127.0.0.1:58816 changed flags via RPC: log_inject_latency_ms_mean from '100' to '600'
I20260808 07:03:23.093103 25343 generic_service.cc:175] {username='slave'} at 127.0.0.1:58816 changed flags via RPC: log_inject_latency_ms_stddev from '100' to '0'
I20260808 07:03:23.093382 25343 generic_service.cc:175] {username='slave'} at 127.0.0.1:58816 changed flags via RPC: log_inject_latency from 'false' to 'true'
I20260808 07:03:23.093544 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 25352
I20260808 07:03:23.101465 16362 test_util.cc:268] Using random seed: -1216089834
I20260808 07:03:23.107371 16362 test_workload.cc:401] TestWorkload: Skipping table creation because table TestTable already exists
W20260808 07:03:23.112339 25643 log.cc:508] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1: Injecting 1200ms of latency in SegmentAllocator::Sync()
W20260808 07:03:23.112557 25501 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.250.130:38349: connect: Connection refused (error 111)
W20260808 07:03:23.112973 25650 log.cc:508] T 3b987c80e5dd489382b90f8d23d6975f P 7b73901566a14a56b74acf52ff5d232e: Injecting 600ms of latency in SegmentAllocator::Sync()
W20260808 07:03:23.113226 25501 consensus_peers.cc:487] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1 -> Peer f112c659629f409a812d76076989e924 (127.15.250.130:38349): Couldn't send request to peer f112c659629f409a812d76076989e924. Status: Network error: Client connection negotiation failed: client connection to 127.15.250.130:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 07:03:23.158313 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 2 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:23.159650 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 2 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:23.210883 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 3 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:23.211884 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 3 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:23.263032 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 4 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:23.263805 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 4 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:23.314890 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 5 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:23.315821 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 5 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:23.366704 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 6 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:23.367498 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 6 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:23.418465 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 7 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:23.419416 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 7 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:23.470328 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 8 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:23.471057 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 8 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:23.521947 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 9 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:23.522773 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 9 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:23.573793 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 10 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:23.574760 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 10 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:23.625921 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 11 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:23.626925 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 11 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:23.678125 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 12 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:23.679210 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 12 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:23.730186 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 13 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:23.731014 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 13 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:23.782052 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 14 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:23.782841 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 14 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:23.833829 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 15 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:23.834810 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 15 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:23.885743 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 16 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:23.886678 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 16 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:23.937758 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 17 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:23.938638 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 17 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:23.989656 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 18 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:23.990432 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 18 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:24.041297 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 19 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:24.042263 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 19 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:24.093127 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 20 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:24.093961 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 20 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:24.144959 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 21 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:24.145974 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 21 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
W20260808 07:03:24.152408 25501 consensus_peers.cc:487] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1 -> Peer f112c659629f409a812d76076989e924 (127.15.250.130:38349): Couldn't send request to peer f112c659629f409a812d76076989e924. Status: Network error: Client connection negotiation failed: client connection to 127.15.250.130:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260808 07:03:24.196874 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 22 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:24.197706 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 22 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:24.248684 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 23 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:24.249512 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 23 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:24.300575 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 24 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:24.301467 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 24 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:24.313021 25663 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:40412 (ReqId={client: 74262b706fc04fe28bc781f8efd4466c, seq_no=0, attempt_no=0}) took 1201 ms. Trace:
I20260808 07:03:24.313126 25663 rpcz_store.cc:276] 0808 07:03:23.111660 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:03:23.111684 (+ 24us) service_pool.cc:227] Handling call
0808 07:03:24.313010 (+1201326us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:03:23.111930 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3b987c80e5dd489382b90f8d23d6975f
0808 07:03:23.111941 (+ 11us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:03:23.111941 (+ 0us) write_op.cc:435] Acquired schema lock
0808 07:03:23.111942 (+ 1us) tablet.cc:660] Decoding operations
0808 07:03:23.111952 (+ 10us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:03:23.111956 (+ 4us) write_op.cc:641] Partition lock acquired for write op
0808 07:03:23.111957 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:03:23.111965 (+ 8us) tablet.cc:699] Row locks acquired
0808 07:03:23.111966 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:03:23.111983 (+ 17us) write_op.cc:270] Start()
0808 07:03:23.111987 (+ 4us) write_op.cc:276] Timestamp: P: 1786172603111937 usec, L: 0
0808 07:03:23.111987 (+ 0us) op_driver.cc:331] REPLICATION: starting
0808 07:03:23.112060 (+ 73us) log.cc:842] Serialized 345 byte log entry
0808 07:03:24.312668 (+1200608us) op_driver.cc:444] REPLICATION: finished
0808 07:03:24.312813 (+ 145us) write_op.cc:301] APPLY: starting
0808 07:03:24.312818 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:03:24.312821 (+ 3us) tablet.cc:1368] finished BulkCheckPresence
0808 07:03:24.312823 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:03:24.312838 (+ 15us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:03:24.312842 (+ 4us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:03:24.312843 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:03:24.312863 (+ 20us) log.cc:842] Serialized 22 byte log entry
0808 07:03:24.312930 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:03:24.312935 (+ 5us) write_op.cc:454] Released schema lock
0808 07:03:24.312935 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":130,"num_ops":1,"prepare.queue_time_us":37,"prepare.run_cpu_time_us":206,"prepare.run_wall_time_us":205,"raft.queue_time_us":89,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":85,"replication_time_us":1200676,"thread_start_us":68,"threads_started":1}]]}
I20260808 07:03:24.352456 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 25 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:24.353175 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 25 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:24.404019 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 26 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:24.404806 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 26 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:24.455691 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 27 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:24.456461 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 27 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:24.507326 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 28 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:24.508214 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 28 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
W20260808 07:03:24.515009 25643 log.cc:508] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1: Injecting 1200ms of latency in SegmentAllocator::Sync()
W20260808 07:03:24.515522 25650 log.cc:508] T 3b987c80e5dd489382b90f8d23d6975f P 7b73901566a14a56b74acf52ff5d232e: Injecting 600ms of latency in SegmentAllocator::Sync()
I20260808 07:03:24.559074 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 29 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:24.559839 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 29 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:24.610678 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 30 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:24.611514 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 30 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:24.662405 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 31 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:24.663308 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 31 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:24.714273 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 32 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:24.715119 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 32 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:24.765980 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 33 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:24.766718 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 33 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:24.817533 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 34 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:24.818310 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 34 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:24.869172 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 35 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:24.869995 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 35 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:24.920893 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 36 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:24.921627 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 36 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:24.972497 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 37 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:24.973251 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 37 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:25.024133 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 38 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:25.024899 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 38 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:25.075834 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 39 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:25.076761 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 39 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:25.127795 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 40 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:25.128827 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 40 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
W20260808 07:03:25.148922 25501 consensus_peers.cc:487] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1 -> Peer f112c659629f409a812d76076989e924 (127.15.250.130:38349): Couldn't send request to peer f112c659629f409a812d76076989e924. Status: Network error: Client connection negotiation failed: client connection to 127.15.250.130:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260808 07:03:25.179791 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 41 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:25.180648 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 41 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:25.231545 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 42 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:25.232390 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 42 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:25.283245 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 43 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:25.284001 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 43 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:25.334966 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 44 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:25.335757 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 44 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:25.386804 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 45 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:25.387737 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 45 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:25.438675 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 46 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:25.439517 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 46 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:25.490504 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 47 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:25.491371 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 47 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:25.542295 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 48 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:25.543090 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 48 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:25.593942 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 49 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:25.594641 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 49 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:25.645516 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 50 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:25.646433 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 50 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:25.697345 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 51 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:25.698288 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 51 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:25.715651 25669 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:40412 (ReqId={client: 74262b706fc04fe28bc781f8efd4466c, seq_no=1, attempt_no=0}) took 1201 ms. Trace:
I20260808 07:03:25.715767 25669 rpcz_store.cc:276] 0808 07:03:24.514389 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:03:24.514446 (+ 57us) service_pool.cc:227] Handling call
0808 07:03:25.715626 (+1201180us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:03:24.514757 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3b987c80e5dd489382b90f8d23d6975f
0808 07:03:24.514769 (+ 12us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:03:24.514770 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:03:24.514770 (+ 0us) tablet.cc:660] Decoding operations
0808 07:03:24.514782 (+ 12us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:03:24.514785 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:03:24.514787 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:03:24.514796 (+ 9us) tablet.cc:699] Row locks acquired
0808 07:03:24.514796 (+ 0us) write_op.cc:260] PREPARE: finished
0808 07:03:24.514804 (+ 8us) write_op.cc:270] Start()
0808 07:03:24.514808 (+ 4us) write_op.cc:276] Timestamp: P: 1786172604514780 usec, L: 0
0808 07:03:24.514808 (+ 0us) op_driver.cc:331] REPLICATION: starting
0808 07:03:24.514848 (+ 40us) log.cc:842] Serialized 354 byte log entry
0808 07:03:25.715319 (+1200471us) op_driver.cc:444] REPLICATION: finished
0808 07:03:25.715458 (+ 139us) write_op.cc:301] APPLY: starting
0808 07:03:25.715463 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:03:25.715466 (+ 3us) tablet.cc:1368] finished BulkCheckPresence
0808 07:03:25.715470 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:03:25.715483 (+ 13us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:03:25.715488 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:03:25.715489 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:03:25.715510 (+ 21us) log.cc:842] Serialized 22 byte log entry
0808 07:03:25.715591 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:03:25.715595 (+ 4us) write_op.cc:454] Released schema lock
0808 07:03:25.715596 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":124,"mutex_wait_us":29,"num_ops":1,"prepare.queue_time_us":105,"prepare.run_cpu_time_us":162,"prepare.run_wall_time_us":162,"raft.queue_time_us":98,"raft.run_cpu_time_us":68,"raft.run_wall_time_us":69,"replication_time_us":1200505,"spinlock_wait_cycles":1664,"thread_start_us":140,"threads_started":2}]]}
I20260808 07:03:25.749258 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 52 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:25.750190 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 52 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:25.801064 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 53 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:25.801927 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 53 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:25.852908 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 54 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:25.853760 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 54 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:25.904634 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 55 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:25.905404 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 55 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
W20260808 07:03:25.917328 25643 log.cc:508] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1: Injecting 1200ms of latency in SegmentAllocator::Sync()
W20260808 07:03:25.917852 25650 log.cc:508] T 3b987c80e5dd489382b90f8d23d6975f P 7b73901566a14a56b74acf52ff5d232e: Injecting 600ms of latency in SegmentAllocator::Sync()
I20260808 07:03:25.956315 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 56 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:25.957213 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 56 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:26.008095 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 57 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:26.008970 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 57 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:26.059980 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 58 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:26.060842 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 58 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:26.111877 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 59 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:26.112756 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 59 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
W20260808 07:03:26.131963 25501 consensus_peers.cc:487] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1 -> Peer f112c659629f409a812d76076989e924 (127.15.250.130:38349): Couldn't send request to peer f112c659629f409a812d76076989e924. Status: Network error: Client connection negotiation failed: client connection to 127.15.250.130:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260808 07:03:26.163700 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 60 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:26.164562 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 60 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:26.215525 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 61 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:26.216427 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 61 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:26.267409 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 62 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:26.268405 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 62 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:26.319315 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 63 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:26.320124 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 63 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:26.370944 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 64 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:26.371819 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 64 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:26.422665 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 65 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:26.423552 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 65 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:26.474447 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 66 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:26.475262 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 66 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:26.526230 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 67 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:26.527098 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 67 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:26.577987 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 68 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:26.578816 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 68 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:26.629868 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 69 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:26.630752 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 69 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:26.681769 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 70 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:26.682605 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 70 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:26.733575 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 71 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:26.734548 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 71 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:26.785494 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 72 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:26.786379 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 72 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:26.837365 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 73 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:26.838281 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 73 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:26.889233 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 74 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:26.890142 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 74 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:26.941046 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 75 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:26.941896 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 75 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
W20260808 07:03:26.962199 25501 consensus_peers.cc:487] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1 -> Peer f112c659629f409a812d76076989e924 (127.15.250.130:38349): Couldn't send request to peer f112c659629f409a812d76076989e924. Status: Network error: Client connection negotiation failed: client connection to 127.15.250.130:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260808 07:03:26.992818 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 76 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:26.993677 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 76 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:27.044651 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 77 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:27.045569 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 77 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:27.096478 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 78 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:27.097249 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 78 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:27.118139 25675 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:40412 (ReqId={client: 74262b706fc04fe28bc781f8efd4466c, seq_no=2, attempt_no=0}) took 1201 ms. Trace:
I20260808 07:03:27.118245 25675 rpcz_store.cc:276] 0808 07:03:25.916832 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:03:25.916872 (+ 40us) service_pool.cc:227] Handling call
0808 07:03:27.118123 (+1201251us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:03:25.917087 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3b987c80e5dd489382b90f8d23d6975f
0808 07:03:25.917100 (+ 13us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:03:25.917101 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:03:25.917102 (+ 1us) tablet.cc:660] Decoding operations
0808 07:03:25.917114 (+ 12us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:03:25.917117 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:03:25.917118 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:03:25.917126 (+ 8us) tablet.cc:699] Row locks acquired
0808 07:03:25.917127 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:03:25.917134 (+ 7us) write_op.cc:270] Start()
0808 07:03:25.917138 (+ 4us) write_op.cc:276] Timestamp: P: 1786172605917061 usec, L: 0
0808 07:03:25.917138 (+ 0us) op_driver.cc:331] REPLICATION: starting
0808 07:03:25.917176 (+ 38us) log.cc:842] Serialized 354 byte log entry
0808 07:03:27.117731 (+1200555us) op_driver.cc:444] REPLICATION: finished
0808 07:03:27.117897 (+ 166us) write_op.cc:301] APPLY: starting
0808 07:03:27.117902 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:03:27.117906 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:03:27.117909 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:03:27.117924 (+ 15us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:03:27.117928 (+ 4us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:03:27.117929 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:03:27.117951 (+ 22us) log.cc:842] Serialized 22 byte log entry
0808 07:03:27.118090 (+ 139us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:03:27.118095 (+ 5us) write_op.cc:454] Released schema lock
0808 07:03:27.118096 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":150,"num_ops":1,"prepare.queue_time_us":110,"prepare.run_cpu_time_us":182,"prepare.run_wall_time_us":183,"raft.queue_time_us":117,"raft.run_cpu_time_us":64,"raft.run_wall_time_us":66,"replication_time_us":1200587,"spinlock_wait_cycles":896,"thread_start_us":152,"threads_started":2}]]}
I20260808 07:03:27.148237 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 79 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:27.149243 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 79 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:27.200163 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 80 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:27.201006 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 80 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:27.251981 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 81 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:27.252856 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 81 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:27.303649 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 82 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:27.304489 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 82 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
W20260808 07:03:27.319880 25643 log.cc:508] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1: Injecting 1200ms of latency in SegmentAllocator::Sync()
W20260808 07:03:27.320530 25650 log.cc:508] T 3b987c80e5dd489382b90f8d23d6975f P 7b73901566a14a56b74acf52ff5d232e: Injecting 600ms of latency in SegmentAllocator::Sync()
I20260808 07:03:27.355490 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 83 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:27.356318 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 83 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:27.407236 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 84 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:27.407989 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 84 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:27.458932 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 85 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:27.459743 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 85 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:27.510625 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 86 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:27.511374 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 86 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:27.562335 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 87 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:27.563093 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 87 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:27.614189 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 88 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:27.614970 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 88 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:27.665918 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 89 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:27.666909 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 89 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:27.718012 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 90 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:27.718746 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 90 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:27.769709 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 91 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:27.770483 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 91 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:27.821470 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 92 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:27.822278 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 92 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
W20260808 07:03:27.829522 25501 consensus_peers.cc:487] T 3b987c80e5dd489382b90f8d23d6975f P 78a3294ee273426aa470db2adbf7f2e1 -> Peer f112c659629f409a812d76076989e924 (127.15.250.130:38349): Couldn't send request to peer f112c659629f409a812d76076989e924. Status: Network error: Client connection negotiation failed: client connection to 127.15.250.130:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260808 07:03:27.873303 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 93 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:27.874118 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 93 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:27.925002 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 94 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:27.925817 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 94 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:27.976802 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 95 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:27.977587 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 95 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:28.028728 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 96 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:28.029662 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 96 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:28.080710 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 97 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:28.081600 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 97 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:28.132714 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 98 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:28.133616 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 98 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:28.184826 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 99 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:28.185678 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 99 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
W20260808 07:03:28.199254 25501 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.250.130:38349: connect: Connection refused (error 111) [suppressed 26 similar messages]
I20260808 07:03:28.236635 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 100 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:28.237432 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 100 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:28.288507 25569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 101 index: 0 } } ignore_live_leader: false dest_uuid: "78a3294ee273426aa470db2adbf7f2e1" is_pre_election: false
I20260808 07:03:28.289458 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b987c80e5dd489382b90f8d23d6975f" candidate_uuid: "f112c659629f409a812d76076989e924" candidate_term: 2 candidate_status { last_received { term: 101 index: 0 } } ignore_live_leader: false dest_uuid: "7b73901566a14a56b74acf52ff5d232e" is_pre_election: false
I20260808 07:03:28.520591 25681 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:40412 (ReqId={client: 74262b706fc04fe28bc781f8efd4466c, seq_no=3, attempt_no=0}) took 1201 ms. Trace:
I20260808 07:03:28.520709 25681 rpcz_store.cc:276] 0808 07:03:27.319308 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 07:03:27.319359 (+ 51us) service_pool.cc:227] Handling call
0808 07:03:28.520553 (+1201194us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 07:03:27.319638 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3b987c80e5dd489382b90f8d23d6975f
0808 07:03:27.319650 (+ 12us) write_op.cc:432] Acquiring schema lock in shared mode
0808 07:03:27.319651 (+ 1us) write_op.cc:435] Acquired schema lock
0808 07:03:27.319651 (+ 0us) tablet.cc:660] Decoding operations
0808 07:03:27.319663 (+ 12us) write_op.cc:620] Acquiring the partition lock for write op
0808 07:03:27.319666 (+ 3us) write_op.cc:641] Partition lock acquired for write op
0808 07:03:27.319667 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0808 07:03:27.319675 (+ 8us) tablet.cc:699] Row locks acquired
0808 07:03:27.319676 (+ 1us) write_op.cc:260] PREPARE: finished
0808 07:03:27.319683 (+ 7us) write_op.cc:270] Start()
0808 07:03:27.319686 (+ 3us) write_op.cc:276] Timestamp: P: 1786172607319641 usec, L: 0
0808 07:03:27.319687 (+ 1us) op_driver.cc:331] REPLICATION: starting
0808 07:03:27.319727 (+ 40us) log.cc:842] Serialized 354 byte log entry
0808 07:03:28.520279 (+1200552us) op_driver.cc:444] REPLICATION: finished
0808 07:03:28.520418 (+ 139us) write_op.cc:301] APPLY: starting
0808 07:03:28.520423 (+ 5us) tablet.cc:1365] starting BulkCheckPresence
0808 07:03:28.520427 (+ 4us) tablet.cc:1368] finished BulkCheckPresence
0808 07:03:28.520430 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 07:03:28.520444 (+ 14us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 07:03:28.520449 (+ 5us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 07:03:28.520450 (+ 1us) write_op.cc:312] APPLY: finished
0808 07:03:28.520472 (+ 22us) log.cc:842] Serialized 22 byte log entry
0808 07:03:28.520516 (+ 44us) write_op.cc:489] Releasing partition, row and schema locks
0808 07:03:28.520520 (+ 4us) write_op.cc:454] Released schema lock
0808 07:03:28.520521 (+ 1us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":124,"num_ops":1,"prepare.queue_time_us":129,"prepare.run_cpu_time_us":268,"prepare.run_wall_time_us":270,"raft.queue_time_us":292,"raft.run_cpu_time_us":75,"raft.run_wall_time_us":76,"replication_time_us":1200586,"spinlock_wait_cycles":13312,"thread_start_us":146,"threads_started":2}]]}
I20260808 07:03:28.721745 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 25217
I20260808 07:03:28.727669 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 25485
I20260808 07:03:28.732944 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 25124
2026-08-08T07:03:28Z chronyd exiting
[ OK ] RaftConsensusElectionITest.DisruptiveServerAndSlowWAL (6467 ms)
[----------] 12 tests from RaftConsensusElectionITest (162251 ms total)
[----------] 2 tests from NumReplicas/RaftConsensusNumLeadersMetricTest
[ RUN ] NumReplicas/RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric/0
I20260808 07:03:28.754308 16362 test_util.cc:268] Using random seed: -1210436993
I20260808 07:03:28.754858 16362 ts_itest-base.cc:112] Starting cluster with:
I20260808 07:03:28.754900 16362 ts_itest-base.cc:113] --------------
I20260808 07:03:28.754911 16362 ts_itest-base.cc:114] 1 tablet servers
I20260808 07:03:28.754922 16362 ts_itest-base.cc:115] 3 replicas per TS
I20260808 07:03:28.754932 16362 ts_itest-base.cc:116] --------------
2026-08-08T07:03:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:03:28Z Disabled control of system clock
I20260808 07:03:28.764525 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.250.190:46817
--webserver_interface=127.15.250.190
--webserver_port=0
--builtin_ntp_servers=127.15.250.148:37459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.250.190:46817
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20260808 07:03:28.847234 25693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:28.847476 25693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:28.847512 25693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:28.848733 25693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:03:28.848798 25693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:28.848824 25693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:03:28.848841 25693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:03:28.850236 25693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:37459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.15.250.190:46817
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.250.190:46817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.250.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.25693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:28.850540 25693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:28.850867 25693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:28.856086 25700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:28.856238 25702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:28.856086 25699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:28.856359 25693 server_base.cc:1034] running on GCE node
I20260808 07:03:28.856678 25693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:28.857016 25693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:28.858183 25693 hybrid_clock.cc:648] HybridClock initialized: now 1786172608858174 us; error 34 us; skew 500 ppm
I20260808 07:03:28.859618 25693 webserver.cc:460] Webserver started at http://127.15.250.190:34523/ using document root <none> and password file <none>
I20260808 07:03:28.859892 25693 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:28.859962 25693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:28.860116 25693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:28.861215 25693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f5e896e1a3b74a4f85eefc7a9fe484c3"
format_stamp: "Formatted at 2026-08-08 07:03:28 on dist-test-slave-0b58"
I20260808 07:03:28.861589 25693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f5e896e1a3b74a4f85eefc7a9fe484c3"
format_stamp: "Formatted at 2026-08-08 07:03:28 on dist-test-slave-0b58"
I20260808 07:03:28.862960 25693 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 07:03:28.863722 25708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:28.863896 25693 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:03:28.863998 25693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f5e896e1a3b74a4f85eefc7a9fe484c3"
format_stamp: "Formatted at 2026-08-08 07:03:28 on dist-test-slave-0b58"
I20260808 07:03:28.864078 25693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:28.886500 25693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:28.886850 25693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:28.887008 25693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:28.891163 25693 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.190:46817
I20260808 07:03:28.891206 25761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.190:46817 every 8 connection(s)
I20260808 07:03:28.891592 25693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260808 07:03:28.892278 25762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:28.894914 25762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5e896e1a3b74a4f85eefc7a9fe484c3: Bootstrap starting.
I20260808 07:03:28.895571 25762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5e896e1a3b74a4f85eefc7a9fe484c3: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:28.895874 25762 log.cc:824] T 00000000000000000000000000000000 P f5e896e1a3b74a4f85eefc7a9fe484c3: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:28.896708 25762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5e896e1a3b74a4f85eefc7a9fe484c3: No bootstrap required, opened a new log
I20260808 07:03:28.898307 25762 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5e896e1a3b74a4f85eefc7a9fe484c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e896e1a3b74a4f85eefc7a9fe484c3" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 46817 } }
I20260808 07:03:28.898450 25762 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5e896e1a3b74a4f85eefc7a9fe484c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:03:28.898473 25762 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5e896e1a3b74a4f85eefc7a9fe484c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e896e1a3b74a4f85eefc7a9fe484c3, State: Initialized, Role: FOLLOWER
I20260808 07:03:28.898643 25762 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5e896e1a3b74a4f85eefc7a9fe484c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e896e1a3b74a4f85eefc7a9fe484c3" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 46817 } }
I20260808 07:03:28.898717 25762 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5e896e1a3b74a4f85eefc7a9fe484c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:03:28.898741 25762 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5e896e1a3b74a4f85eefc7a9fe484c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:03:28.898769 25762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5e896e1a3b74a4f85eefc7a9fe484c3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:28.899338 25762 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5e896e1a3b74a4f85eefc7a9fe484c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e896e1a3b74a4f85eefc7a9fe484c3" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 46817 } }
I20260808 07:03:28.899454 25762 leader_election.cc:302] T 00000000000000000000000000000000 P f5e896e1a3b74a4f85eefc7a9fe484c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5e896e1a3b74a4f85eefc7a9fe484c3; no voters:
I20260808 07:03:28.899627 25762 leader_election.cc:288] T 00000000000000000000000000000000 P f5e896e1a3b74a4f85eefc7a9fe484c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:03:28.899694 25765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5e896e1a3b74a4f85eefc7a9fe484c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:28.899791 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 25693
I20260808 07:03:28.899861 25765 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5e896e1a3b74a4f85eefc7a9fe484c3 [term 1 LEADER]: Becoming Leader. State: Replica: f5e896e1a3b74a4f85eefc7a9fe484c3, State: Running, Role: LEADER
I20260808 07:03:28.899894 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260808 07:03:28.899932 25762 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5e896e1a3b74a4f85eefc7a9fe484c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:03:28.899945 25765 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5e896e1a3b74a4f85eefc7a9fe484c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e896e1a3b74a4f85eefc7a9fe484c3" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 46817 } }
I20260808 07:03:28.900357 25766 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5e896e1a3b74a4f85eefc7a9fe484c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5e896e1a3b74a4f85eefc7a9fe484c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e896e1a3b74a4f85eefc7a9fe484c3" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 46817 } } }
I20260808 07:03:28.900436 25766 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5e896e1a3b74a4f85eefc7a9fe484c3 [sys.catalog]: This master's current role is: LEADER
I20260808 07:03:28.900688 25767 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5e896e1a3b74a4f85eefc7a9fe484c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5e896e1a3b74a4f85eefc7a9fe484c3. Latest consensus state: current_term: 1 leader_uuid: "f5e896e1a3b74a4f85eefc7a9fe484c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e896e1a3b74a4f85eefc7a9fe484c3" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 46817 } } }
I20260808 07:03:28.901041 25767 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5e896e1a3b74a4f85eefc7a9fe484c3 [sys.catalog]: This master's current role is: LEADER
I20260808 07:03:28.901314 25771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:03:28.901919 25771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:03:28.903610 25771 catalog_manager.cc:1347] Generated new cluster ID: cd1abd83eaad4af49721e4e32d549a5a
I20260808 07:03:28.903666 25771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:03:28.917613 25771 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:03:28.918360 25771 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:03:28.928925 25771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5e896e1a3b74a4f85eefc7a9fe484c3: Generated new TSK 0
I20260808 07:03:28.929175 25771 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:03:28.930769 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.129:0
--local_ip_for_outbound_sockets=127.15.250.129
--webserver_interface=127.15.250.129
--webserver_port=0
--tserver_master_addrs=127.15.250.190:46817
--builtin_ntp_servers=127.15.250.148:37459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20260808 07:03:29.016749 25788 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 07:03:29.016959 25788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:29.016975 25788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:29.016994 25788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:29.018178 25788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:29.018234 25788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.129
I20260808 07:03:29.019591 25788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:37459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.250.129
--webserver_port=0
--tserver_master_addrs=127.15.250.190:46817
--never_fsync=true
--heap_profile_path=/tmp/kudu.25788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.129
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:29.019855 25788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:29.020198 25788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:29.026473 25794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:29.026531 25788 server_base.cc:1034] running on GCE node
W20260808 07:03:29.026610 25793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:29.026481 25796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:29.026883 25788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:29.027153 25788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:29.028359 25788 hybrid_clock.cc:648] HybridClock initialized: now 1786172609028337 us; error 36 us; skew 500 ppm
I20260808 07:03:29.029748 25788 webserver.cc:460] Webserver started at http://127.15.250.129:39861/ using document root <none> and password file <none>
I20260808 07:03:29.030014 25788 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:29.030066 25788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:29.030236 25788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:29.031312 25788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0"
format_stamp: "Formatted at 2026-08-08 07:03:29 on dist-test-slave-0b58"
I20260808 07:03:29.031693 25788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0"
format_stamp: "Formatted at 2026-08-08 07:03:29 on dist-test-slave-0b58"
I20260808 07:03:29.033246 25788 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 07:03:29.034044 25802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.034260 25788 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:03:29.034406 25788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0"
format_stamp: "Formatted at 2026-08-08 07:03:29 on dist-test-slave-0b58"
I20260808 07:03:29.034493 25788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:29.044135 25788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:29.044492 25788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:29.044642 25788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:29.044883 25788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:03:29.045238 25788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:03:29.045297 25788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.045341 25788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:03:29.045368 25788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.051366 25788 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.129:45633
I20260808 07:03:29.051443 25922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.129:45633 every 8 connection(s)
I20260808 07:03:29.051795 25788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260808 07:03:29.056756 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 25788
I20260808 07:03:29.056874 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260808 07:03:29.058036 25923 heartbeater.cc:347] Connected to a master server at 127.15.250.190:46817
I20260808 07:03:29.058135 25923 heartbeater.cc:464] Registering TS with master...
I20260808 07:03:29.058331 25923 heartbeater.cc:511] Master 127.15.250.190:46817 requested a full tablet report, sending...
I20260808 07:03:29.058813 25725 ts_manager.cc:194] Registered new tserver with Master: 1bb61bde5c6d4e73a41f0f7a1e7777c0 (127.15.250.129:45633)
I20260808 07:03:29.059473 25725 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.129:49843
I20260808 07:03:29.069175 16362 external_mini_cluster.cc:930] 1 TS(s) registered with all masters
I20260808 07:03:29.076006 16362 external_mini_cluster.cc:930] 1 TS(s) registered with all masters
I20260808 07:03:29.076081 16362 test_util.cc:268] Using random seed: -1210115216
I20260808 07:03:29.083384 25725 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49520:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 07:03:29.090636 25851 tablet_service.cc:1467] Processing CreateTablet for tablet f24fedb649434547bc1d2f58f50bc963 (DEFAULT_TABLE table=test-workload [id=d3caa4ae12d642b499d2a568594bcd3d]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260808 07:03:29.090656 25852 tablet_service.cc:1467] Processing CreateTablet for tablet 47c5d39ba24e45e0a148e9c8dc92a10d (DEFAULT_TABLE table=test-workload [id=d3caa4ae12d642b499d2a568594bcd3d]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260808 07:03:29.090945 25844 tablet_service.cc:1467] Processing CreateTablet for tablet 61c600f41d1e4e158c5e9f112bb448f2 (DEFAULT_TABLE table=test-workload [id=d3caa4ae12d642b499d2a568594bcd3d]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260808 07:03:29.090636 25854 tablet_service.cc:1467] Processing CreateTablet for tablet 0ad895cdfa404ab5bac618ca14edde34 (DEFAULT_TABLE table=test-workload [id=d3caa4ae12d642b499d2a568594bcd3d]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260808 07:03:29.091358 25850 tablet_service.cc:1467] Processing CreateTablet for tablet de6e1931655b4f049fcfffa9039de87c (DEFAULT_TABLE table=test-workload [id=d3caa4ae12d642b499d2a568594bcd3d]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260808 07:03:29.091248 25844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61c600f41d1e4e158c5e9f112bb448f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.090689 25848 tablet_service.cc:1467] Processing CreateTablet for tablet 619e3f62e1cc462dae57b054f2af0236 (DEFAULT_TABLE table=test-workload [id=d3caa4ae12d642b499d2a568594bcd3d]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260808 07:03:29.090809 25845 tablet_service.cc:1467] Processing CreateTablet for tablet 6922517948894072b8bec2b418cc0bb5 (DEFAULT_TABLE table=test-workload [id=d3caa4ae12d642b499d2a568594bcd3d]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260808 07:03:29.091522 25852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47c5d39ba24e45e0a148e9c8dc92a10d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.091609 25845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6922517948894072b8bec2b418cc0bb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.090636 25853 tablet_service.cc:1467] Processing CreateTablet for tablet 681fdba8b3b2422cb38e1e6bb20c5a60 (DEFAULT_TABLE table=test-workload [id=d3caa4ae12d642b499d2a568594bcd3d]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260808 07:03:29.091747 25851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f24fedb649434547bc1d2f58f50bc963. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.091843 25853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 681fdba8b3b2422cb38e1e6bb20c5a60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.090636 25849 tablet_service.cc:1467] Processing CreateTablet for tablet 1b76850c4e9349f0b1c95093f15d21c1 (DEFAULT_TABLE table=test-workload [id=d3caa4ae12d642b499d2a568594bcd3d]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260808 07:03:29.092119 25849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b76850c4e9349f0b1c95093f15d21c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.092185 25848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 619e3f62e1cc462dae57b054f2af0236. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.092278 25850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de6e1931655b4f049fcfffa9039de87c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.092494 25854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ad895cdfa404ab5bac618ca14edde34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.090773 25847 tablet_service.cc:1467] Processing CreateTablet for tablet 9f629ac47ce046f5b650998fcb743ad0 (DEFAULT_TABLE table=test-workload [id=d3caa4ae12d642b499d2a568594bcd3d]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260808 07:03:29.093109 25847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f629ac47ce046f5b650998fcb743ad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.095202 25938 tablet_bootstrap.cc:492] T 9f629ac47ce046f5b650998fcb743ad0 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Bootstrap starting.
I20260808 07:03:29.095938 25938 tablet_bootstrap.cc:654] T 9f629ac47ce046f5b650998fcb743ad0 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.096325 25938 log.cc:824] T 9f629ac47ce046f5b650998fcb743ad0 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:29.097078 25938 tablet_bootstrap.cc:492] T 9f629ac47ce046f5b650998fcb743ad0 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: No bootstrap required, opened a new log
I20260808 07:03:29.097136 25938 ts_tablet_manager.cc:1397] T 9f629ac47ce046f5b650998fcb743ad0 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:03:29.098965 25938 raft_consensus.cc:348] T 9f629ac47ce046f5b650998fcb743ad0 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.099140 25938 raft_consensus.cc:729] T 9f629ac47ce046f5b650998fcb743ad0 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bb61bde5c6d4e73a41f0f7a1e7777c0, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.099236 25938 consensus_queue.cc:260] T 9f629ac47ce046f5b650998fcb743ad0 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.099576 25938 ts_tablet_manager.cc:1428] T 9f629ac47ce046f5b650998fcb743ad0 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 07:03:29.099601 25923 heartbeater.cc:503] Master 127.15.250.190:46817 was elected leader, sending a full tablet report...
I20260808 07:03:29.099686 25938 tablet_bootstrap.cc:492] T de6e1931655b4f049fcfffa9039de87c P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Bootstrap starting.
I20260808 07:03:29.100142 25938 tablet_bootstrap.cc:654] T de6e1931655b4f049fcfffa9039de87c P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.100899 25938 tablet_bootstrap.cc:492] T de6e1931655b4f049fcfffa9039de87c P 1bb61bde5c6d4e73a41f0f7a1e7777c0: No bootstrap required, opened a new log
I20260808 07:03:29.100970 25938 ts_tablet_manager.cc:1397] T de6e1931655b4f049fcfffa9039de87c P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 07:03:29.101140 25938 raft_consensus.cc:348] T de6e1931655b4f049fcfffa9039de87c P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.101228 25938 raft_consensus.cc:729] T de6e1931655b4f049fcfffa9039de87c P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bb61bde5c6d4e73a41f0f7a1e7777c0, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.101257 25938 consensus_queue.cc:260] T de6e1931655b4f049fcfffa9039de87c P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.101344 25938 ts_tablet_manager.cc:1428] T de6e1931655b4f049fcfffa9039de87c P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.101403 25938 tablet_bootstrap.cc:492] T 6922517948894072b8bec2b418cc0bb5 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Bootstrap starting.
I20260808 07:03:29.101789 25938 tablet_bootstrap.cc:654] T 6922517948894072b8bec2b418cc0bb5 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.102453 25938 tablet_bootstrap.cc:492] T 6922517948894072b8bec2b418cc0bb5 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: No bootstrap required, opened a new log
I20260808 07:03:29.102512 25938 ts_tablet_manager.cc:1397] T 6922517948894072b8bec2b418cc0bb5 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 07:03:29.102689 25938 raft_consensus.cc:348] T 6922517948894072b8bec2b418cc0bb5 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.102763 25938 raft_consensus.cc:729] T 6922517948894072b8bec2b418cc0bb5 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bb61bde5c6d4e73a41f0f7a1e7777c0, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.102793 25938 consensus_queue.cc:260] T 6922517948894072b8bec2b418cc0bb5 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.102907 25938 ts_tablet_manager.cc:1428] T 6922517948894072b8bec2b418cc0bb5 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.102980 25938 tablet_bootstrap.cc:492] T 47c5d39ba24e45e0a148e9c8dc92a10d P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Bootstrap starting.
I20260808 07:03:29.103343 25938 tablet_bootstrap.cc:654] T 47c5d39ba24e45e0a148e9c8dc92a10d P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.104310 25938 tablet_bootstrap.cc:492] T 47c5d39ba24e45e0a148e9c8dc92a10d P 1bb61bde5c6d4e73a41f0f7a1e7777c0: No bootstrap required, opened a new log
I20260808 07:03:29.104363 25938 ts_tablet_manager.cc:1397] T 47c5d39ba24e45e0a148e9c8dc92a10d P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 07:03:29.104463 25938 raft_consensus.cc:348] T 47c5d39ba24e45e0a148e9c8dc92a10d P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.104509 25938 raft_consensus.cc:729] T 47c5d39ba24e45e0a148e9c8dc92a10d P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bb61bde5c6d4e73a41f0f7a1e7777c0, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.104527 25938 consensus_queue.cc:260] T 47c5d39ba24e45e0a148e9c8dc92a10d P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.104610 25938 ts_tablet_manager.cc:1428] T 47c5d39ba24e45e0a148e9c8dc92a10d P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.104697 25938 tablet_bootstrap.cc:492] T 619e3f62e1cc462dae57b054f2af0236 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Bootstrap starting.
I20260808 07:03:29.105113 25938 tablet_bootstrap.cc:654] T 619e3f62e1cc462dae57b054f2af0236 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.105765 25938 tablet_bootstrap.cc:492] T 619e3f62e1cc462dae57b054f2af0236 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: No bootstrap required, opened a new log
I20260808 07:03:29.105809 25938 ts_tablet_manager.cc:1397] T 619e3f62e1cc462dae57b054f2af0236 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 07:03:29.105930 25938 raft_consensus.cc:348] T 619e3f62e1cc462dae57b054f2af0236 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.105975 25938 raft_consensus.cc:729] T 619e3f62e1cc462dae57b054f2af0236 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bb61bde5c6d4e73a41f0f7a1e7777c0, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.105996 25938 consensus_queue.cc:260] T 619e3f62e1cc462dae57b054f2af0236 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.106079 25938 ts_tablet_manager.cc:1428] T 619e3f62e1cc462dae57b054f2af0236 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.106144 25938 tablet_bootstrap.cc:492] T 681fdba8b3b2422cb38e1e6bb20c5a60 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Bootstrap starting.
I20260808 07:03:29.106554 25938 tablet_bootstrap.cc:654] T 681fdba8b3b2422cb38e1e6bb20c5a60 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.107239 25938 tablet_bootstrap.cc:492] T 681fdba8b3b2422cb38e1e6bb20c5a60 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: No bootstrap required, opened a new log
I20260808 07:03:29.107286 25938 ts_tablet_manager.cc:1397] T 681fdba8b3b2422cb38e1e6bb20c5a60 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 07:03:29.107400 25938 raft_consensus.cc:348] T 681fdba8b3b2422cb38e1e6bb20c5a60 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.107461 25938 raft_consensus.cc:729] T 681fdba8b3b2422cb38e1e6bb20c5a60 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bb61bde5c6d4e73a41f0f7a1e7777c0, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.107481 25938 consensus_queue.cc:260] T 681fdba8b3b2422cb38e1e6bb20c5a60 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.107560 25938 ts_tablet_manager.cc:1428] T 681fdba8b3b2422cb38e1e6bb20c5a60 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.107618 25938 tablet_bootstrap.cc:492] T 61c600f41d1e4e158c5e9f112bb448f2 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Bootstrap starting.
I20260808 07:03:29.108007 25938 tablet_bootstrap.cc:654] T 61c600f41d1e4e158c5e9f112bb448f2 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.108839 25938 tablet_bootstrap.cc:492] T 61c600f41d1e4e158c5e9f112bb448f2 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: No bootstrap required, opened a new log
I20260808 07:03:29.108880 25938 ts_tablet_manager.cc:1397] T 61c600f41d1e4e158c5e9f112bb448f2 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260808 07:03:29.109010 25938 raft_consensus.cc:348] T 61c600f41d1e4e158c5e9f112bb448f2 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.109064 25938 raft_consensus.cc:729] T 61c600f41d1e4e158c5e9f112bb448f2 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bb61bde5c6d4e73a41f0f7a1e7777c0, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.109083 25938 consensus_queue.cc:260] T 61c600f41d1e4e158c5e9f112bb448f2 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.109195 25938 ts_tablet_manager.cc:1428] T 61c600f41d1e4e158c5e9f112bb448f2 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.109287 25938 tablet_bootstrap.cc:492] T 0ad895cdfa404ab5bac618ca14edde34 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Bootstrap starting.
I20260808 07:03:29.109638 25938 tablet_bootstrap.cc:654] T 0ad895cdfa404ab5bac618ca14edde34 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.110316 25938 tablet_bootstrap.cc:492] T 0ad895cdfa404ab5bac618ca14edde34 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: No bootstrap required, opened a new log
I20260808 07:03:29.110364 25938 ts_tablet_manager.cc:1397] T 0ad895cdfa404ab5bac618ca14edde34 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260808 07:03:29.110507 25938 raft_consensus.cc:348] T 0ad895cdfa404ab5bac618ca14edde34 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.110574 25938 raft_consensus.cc:729] T 0ad895cdfa404ab5bac618ca14edde34 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bb61bde5c6d4e73a41f0f7a1e7777c0, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.110622 25938 consensus_queue.cc:260] T 0ad895cdfa404ab5bac618ca14edde34 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.110710 25938 ts_tablet_manager.cc:1428] T 0ad895cdfa404ab5bac618ca14edde34 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.110765 25938 tablet_bootstrap.cc:492] T f24fedb649434547bc1d2f58f50bc963 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Bootstrap starting.
I20260808 07:03:29.111146 25938 tablet_bootstrap.cc:654] T f24fedb649434547bc1d2f58f50bc963 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.111768 25938 tablet_bootstrap.cc:492] T f24fedb649434547bc1d2f58f50bc963 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: No bootstrap required, opened a new log
I20260808 07:03:29.111828 25938 ts_tablet_manager.cc:1397] T f24fedb649434547bc1d2f58f50bc963 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260808 07:03:29.111958 25938 raft_consensus.cc:348] T f24fedb649434547bc1d2f58f50bc963 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.112012 25938 raft_consensus.cc:729] T f24fedb649434547bc1d2f58f50bc963 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bb61bde5c6d4e73a41f0f7a1e7777c0, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.112030 25938 consensus_queue.cc:260] T f24fedb649434547bc1d2f58f50bc963 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.112126 25938 ts_tablet_manager.cc:1428] T f24fedb649434547bc1d2f58f50bc963 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.112202 25938 tablet_bootstrap.cc:492] T 1b76850c4e9349f0b1c95093f15d21c1 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Bootstrap starting.
I20260808 07:03:29.112617 25938 tablet_bootstrap.cc:654] T 1b76850c4e9349f0b1c95093f15d21c1 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.113322 25938 tablet_bootstrap.cc:492] T 1b76850c4e9349f0b1c95093f15d21c1 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: No bootstrap required, opened a new log
I20260808 07:03:29.113382 25938 ts_tablet_manager.cc:1397] T 1b76850c4e9349f0b1c95093f15d21c1 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 07:03:29.113516 25938 raft_consensus.cc:348] T 1b76850c4e9349f0b1c95093f15d21c1 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.113584 25938 raft_consensus.cc:729] T 1b76850c4e9349f0b1c95093f15d21c1 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bb61bde5c6d4e73a41f0f7a1e7777c0, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.113626 25938 consensus_queue.cc:260] T 1b76850c4e9349f0b1c95093f15d21c1 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.113703 25938 ts_tablet_manager.cc:1428] T 1b76850c4e9349f0b1c95093f15d21c1 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.126065 25875 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "61c600f41d1e4e158c5e9f112bb448f2"
dest_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0"
from {username='slave'} at 127.0.0.1:38226
I20260808 07:03:29.126261 25875 raft_consensus.cc:482] T 61c600f41d1e4e158c5e9f112bb448f2 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 07:03:29.126349 25875 raft_consensus.cc:3037] T 61c600f41d1e4e158c5e9f112bb448f2 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.127035 25875 raft_consensus.cc:504] T 61c600f41d1e4e158c5e9f112bb448f2 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.127220 25875 leader_election.cc:302] T 61c600f41d1e4e158c5e9f112bb448f2 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bb61bde5c6d4e73a41f0f7a1e7777c0; no voters:
I20260808 07:03:29.127333 25875 leader_election.cc:288] T 61c600f41d1e4e158c5e9f112bb448f2 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:03:29.127389 25940 raft_consensus.cc:2781] T 61c600f41d1e4e158c5e9f112bb448f2 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:29.127640 25940 raft_consensus.cc:686] T 61c600f41d1e4e158c5e9f112bb448f2 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 LEADER]: Becoming Leader. State: Replica: 1bb61bde5c6d4e73a41f0f7a1e7777c0, State: Running, Role: LEADER
I20260808 07:03:29.127750 25940 consensus_queue.cc:237] T 61c600f41d1e4e158c5e9f112bb448f2 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.128600 25725 catalog_manager.cc:5495] T 61c600f41d1e4e158c5e9f112bb448f2 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bb61bde5c6d4e73a41f0f7a1e7777c0 (127.15.250.129). New cstate: current_term: 1 leader_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } health_report { overall_health: HEALTHY } } }
I20260808 07:03:29.129693 25875 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9f629ac47ce046f5b650998fcb743ad0"
dest_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0"
from {username='slave'} at 127.0.0.1:38226
I20260808 07:03:29.129787 25875 raft_consensus.cc:482] T 9f629ac47ce046f5b650998fcb743ad0 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 07:03:29.129818 25875 raft_consensus.cc:3037] T 9f629ac47ce046f5b650998fcb743ad0 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.130388 25875 raft_consensus.cc:504] T 9f629ac47ce046f5b650998fcb743ad0 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.130471 25875 leader_election.cc:302] T 9f629ac47ce046f5b650998fcb743ad0 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bb61bde5c6d4e73a41f0f7a1e7777c0; no voters:
I20260808 07:03:29.130527 25875 leader_election.cc:288] T 9f629ac47ce046f5b650998fcb743ad0 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:03:29.130577 25940 raft_consensus.cc:2781] T 9f629ac47ce046f5b650998fcb743ad0 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:29.130648 25940 raft_consensus.cc:686] T 9f629ac47ce046f5b650998fcb743ad0 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 LEADER]: Becoming Leader. State: Replica: 1bb61bde5c6d4e73a41f0f7a1e7777c0, State: Running, Role: LEADER
I20260808 07:03:29.130708 25940 consensus_queue.cc:237] T 9f629ac47ce046f5b650998fcb743ad0 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.131209 25725 catalog_manager.cc:5495] T 9f629ac47ce046f5b650998fcb743ad0 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bb61bde5c6d4e73a41f0f7a1e7777c0 (127.15.250.129). New cstate: current_term: 1 leader_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } health_report { overall_health: HEALTHY } } }
I20260808 07:03:29.132385 25875 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "de6e1931655b4f049fcfffa9039de87c"
dest_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0"
from {username='slave'} at 127.0.0.1:38226
I20260808 07:03:29.132468 25875 raft_consensus.cc:482] T de6e1931655b4f049fcfffa9039de87c P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 07:03:29.132498 25875 raft_consensus.cc:3037] T de6e1931655b4f049fcfffa9039de87c P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.133004 25875 raft_consensus.cc:504] T de6e1931655b4f049fcfffa9039de87c P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.133083 25875 leader_election.cc:302] T de6e1931655b4f049fcfffa9039de87c P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bb61bde5c6d4e73a41f0f7a1e7777c0; no voters:
I20260808 07:03:29.133136 25875 leader_election.cc:288] T de6e1931655b4f049fcfffa9039de87c P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:03:29.133170 25940 raft_consensus.cc:2781] T de6e1931655b4f049fcfffa9039de87c P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:29.133235 25940 raft_consensus.cc:686] T de6e1931655b4f049fcfffa9039de87c P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 LEADER]: Becoming Leader. State: Replica: 1bb61bde5c6d4e73a41f0f7a1e7777c0, State: Running, Role: LEADER
I20260808 07:03:29.133317 25940 consensus_queue.cc:237] T de6e1931655b4f049fcfffa9039de87c P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.133816 25725 catalog_manager.cc:5495] T de6e1931655b4f049fcfffa9039de87c P 1bb61bde5c6d4e73a41f0f7a1e7777c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bb61bde5c6d4e73a41f0f7a1e7777c0 (127.15.250.129). New cstate: current_term: 1 leader_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } health_report { overall_health: HEALTHY } } }
I20260808 07:03:29.135058 25875 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6922517948894072b8bec2b418cc0bb5"
dest_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0"
from {username='slave'} at 127.0.0.1:38226
I20260808 07:03:29.135128 25875 raft_consensus.cc:482] T 6922517948894072b8bec2b418cc0bb5 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 07:03:29.135149 25875 raft_consensus.cc:3037] T 6922517948894072b8bec2b418cc0bb5 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.135618 25875 raft_consensus.cc:504] T 6922517948894072b8bec2b418cc0bb5 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.135697 25875 leader_election.cc:302] T 6922517948894072b8bec2b418cc0bb5 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bb61bde5c6d4e73a41f0f7a1e7777c0; no voters:
I20260808 07:03:29.135747 25875 leader_election.cc:288] T 6922517948894072b8bec2b418cc0bb5 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:03:29.135838 25952 raft_consensus.cc:2781] T 6922517948894072b8bec2b418cc0bb5 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:29.135888 25952 raft_consensus.cc:686] T 6922517948894072b8bec2b418cc0bb5 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 LEADER]: Becoming Leader. State: Replica: 1bb61bde5c6d4e73a41f0f7a1e7777c0, State: Running, Role: LEADER
I20260808 07:03:29.135933 25952 consensus_queue.cc:237] T 6922517948894072b8bec2b418cc0bb5 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.136492 25725 catalog_manager.cc:5495] T 6922517948894072b8bec2b418cc0bb5 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bb61bde5c6d4e73a41f0f7a1e7777c0 (127.15.250.129). New cstate: current_term: 1 leader_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } health_report { overall_health: HEALTHY } } }
I20260808 07:03:29.137570 25875 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0ad895cdfa404ab5bac618ca14edde34"
dest_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0"
from {username='slave'} at 127.0.0.1:38226
I20260808 07:03:29.137627 25875 raft_consensus.cc:482] T 0ad895cdfa404ab5bac618ca14edde34 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 07:03:29.137648 25875 raft_consensus.cc:3037] T 0ad895cdfa404ab5bac618ca14edde34 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.138144 25875 raft_consensus.cc:504] T 0ad895cdfa404ab5bac618ca14edde34 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.138221 25875 leader_election.cc:302] T 0ad895cdfa404ab5bac618ca14edde34 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bb61bde5c6d4e73a41f0f7a1e7777c0; no voters:
I20260808 07:03:29.138270 25875 leader_election.cc:288] T 0ad895cdfa404ab5bac618ca14edde34 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:03:29.138288 25952 raft_consensus.cc:2781] T 0ad895cdfa404ab5bac618ca14edde34 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:29.138415 25952 raft_consensus.cc:686] T 0ad895cdfa404ab5bac618ca14edde34 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 LEADER]: Becoming Leader. State: Replica: 1bb61bde5c6d4e73a41f0f7a1e7777c0, State: Running, Role: LEADER
I20260808 07:03:29.138461 25952 consensus_queue.cc:237] T 0ad895cdfa404ab5bac618ca14edde34 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.138897 25725 catalog_manager.cc:5495] T 0ad895cdfa404ab5bac618ca14edde34 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bb61bde5c6d4e73a41f0f7a1e7777c0 (127.15.250.129). New cstate: current_term: 1 leader_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } health_report { overall_health: HEALTHY } } }
I20260808 07:03:29.140327 25875 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "47c5d39ba24e45e0a148e9c8dc92a10d"
dest_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0"
from {username='slave'} at 127.0.0.1:38226
I20260808 07:03:29.140398 25875 raft_consensus.cc:482] T 47c5d39ba24e45e0a148e9c8dc92a10d P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 07:03:29.140419 25875 raft_consensus.cc:3037] T 47c5d39ba24e45e0a148e9c8dc92a10d P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.140854 25875 raft_consensus.cc:504] T 47c5d39ba24e45e0a148e9c8dc92a10d P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.140916 25875 leader_election.cc:302] T 47c5d39ba24e45e0a148e9c8dc92a10d P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bb61bde5c6d4e73a41f0f7a1e7777c0; no voters:
I20260808 07:03:29.140970 25875 leader_election.cc:288] T 47c5d39ba24e45e0a148e9c8dc92a10d P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:03:29.140987 25952 raft_consensus.cc:2781] T 47c5d39ba24e45e0a148e9c8dc92a10d P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:29.141049 25952 raft_consensus.cc:686] T 47c5d39ba24e45e0a148e9c8dc92a10d P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 LEADER]: Becoming Leader. State: Replica: 1bb61bde5c6d4e73a41f0f7a1e7777c0, State: Running, Role: LEADER
I20260808 07:03:29.141101 25952 consensus_queue.cc:237] T 47c5d39ba24e45e0a148e9c8dc92a10d P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.141562 25725 catalog_manager.cc:5495] T 47c5d39ba24e45e0a148e9c8dc92a10d P 1bb61bde5c6d4e73a41f0f7a1e7777c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bb61bde5c6d4e73a41f0f7a1e7777c0 (127.15.250.129). New cstate: current_term: 1 leader_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } health_report { overall_health: HEALTHY } } }
I20260808 07:03:29.142825 25875 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f24fedb649434547bc1d2f58f50bc963"
dest_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0"
from {username='slave'} at 127.0.0.1:38226
I20260808 07:03:29.142884 25875 raft_consensus.cc:482] T f24fedb649434547bc1d2f58f50bc963 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 07:03:29.142902 25875 raft_consensus.cc:3037] T f24fedb649434547bc1d2f58f50bc963 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.143404 25875 raft_consensus.cc:504] T f24fedb649434547bc1d2f58f50bc963 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.143465 25875 leader_election.cc:302] T f24fedb649434547bc1d2f58f50bc963 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bb61bde5c6d4e73a41f0f7a1e7777c0; no voters:
I20260808 07:03:29.143518 25875 leader_election.cc:288] T f24fedb649434547bc1d2f58f50bc963 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:03:29.143669 25952 raft_consensus.cc:2781] T f24fedb649434547bc1d2f58f50bc963 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:29.143735 25952 raft_consensus.cc:686] T f24fedb649434547bc1d2f58f50bc963 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 LEADER]: Becoming Leader. State: Replica: 1bb61bde5c6d4e73a41f0f7a1e7777c0, State: Running, Role: LEADER
I20260808 07:03:29.143776 25952 consensus_queue.cc:237] T f24fedb649434547bc1d2f58f50bc963 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.144222 25725 catalog_manager.cc:5495] T f24fedb649434547bc1d2f58f50bc963 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bb61bde5c6d4e73a41f0f7a1e7777c0 (127.15.250.129). New cstate: current_term: 1 leader_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } health_report { overall_health: HEALTHY } } }
I20260808 07:03:29.145475 25875 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "619e3f62e1cc462dae57b054f2af0236"
dest_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0"
from {username='slave'} at 127.0.0.1:38226
I20260808 07:03:29.145531 25875 raft_consensus.cc:482] T 619e3f62e1cc462dae57b054f2af0236 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 07:03:29.145551 25875 raft_consensus.cc:3037] T 619e3f62e1cc462dae57b054f2af0236 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.146013 25875 raft_consensus.cc:504] T 619e3f62e1cc462dae57b054f2af0236 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.146075 25875 leader_election.cc:302] T 619e3f62e1cc462dae57b054f2af0236 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bb61bde5c6d4e73a41f0f7a1e7777c0; no voters:
I20260808 07:03:29.146138 25875 leader_election.cc:288] T 619e3f62e1cc462dae57b054f2af0236 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:03:29.146188 25952 raft_consensus.cc:2781] T 619e3f62e1cc462dae57b054f2af0236 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:29.146257 25952 raft_consensus.cc:686] T 619e3f62e1cc462dae57b054f2af0236 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 LEADER]: Becoming Leader. State: Replica: 1bb61bde5c6d4e73a41f0f7a1e7777c0, State: Running, Role: LEADER
I20260808 07:03:29.146303 25952 consensus_queue.cc:237] T 619e3f62e1cc462dae57b054f2af0236 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.146776 25725 catalog_manager.cc:5495] T 619e3f62e1cc462dae57b054f2af0236 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bb61bde5c6d4e73a41f0f7a1e7777c0 (127.15.250.129). New cstate: current_term: 1 leader_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } health_report { overall_health: HEALTHY } } }
I20260808 07:03:29.147799 25875 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1b76850c4e9349f0b1c95093f15d21c1"
dest_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0"
from {username='slave'} at 127.0.0.1:38226
I20260808 07:03:29.147877 25875 raft_consensus.cc:482] T 1b76850c4e9349f0b1c95093f15d21c1 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 07:03:29.147907 25875 raft_consensus.cc:3037] T 1b76850c4e9349f0b1c95093f15d21c1 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.148391 25875 raft_consensus.cc:504] T 1b76850c4e9349f0b1c95093f15d21c1 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.148464 25875 leader_election.cc:302] T 1b76850c4e9349f0b1c95093f15d21c1 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bb61bde5c6d4e73a41f0f7a1e7777c0; no voters:
I20260808 07:03:29.148518 25875 leader_election.cc:288] T 1b76850c4e9349f0b1c95093f15d21c1 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:03:29.148535 25952 raft_consensus.cc:2781] T 1b76850c4e9349f0b1c95093f15d21c1 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:29.148577 25952 raft_consensus.cc:686] T 1b76850c4e9349f0b1c95093f15d21c1 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 LEADER]: Becoming Leader. State: Replica: 1bb61bde5c6d4e73a41f0f7a1e7777c0, State: Running, Role: LEADER
I20260808 07:03:29.148619 25952 consensus_queue.cc:237] T 1b76850c4e9349f0b1c95093f15d21c1 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.149117 25725 catalog_manager.cc:5495] T 1b76850c4e9349f0b1c95093f15d21c1 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bb61bde5c6d4e73a41f0f7a1e7777c0 (127.15.250.129). New cstate: current_term: 1 leader_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } health_report { overall_health: HEALTHY } } }
I20260808 07:03:29.150123 25875 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "681fdba8b3b2422cb38e1e6bb20c5a60"
dest_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0"
from {username='slave'} at 127.0.0.1:38226
I20260808 07:03:29.150182 25875 raft_consensus.cc:482] T 681fdba8b3b2422cb38e1e6bb20c5a60 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 07:03:29.150199 25875 raft_consensus.cc:3037] T 681fdba8b3b2422cb38e1e6bb20c5a60 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.150673 25875 raft_consensus.cc:504] T 681fdba8b3b2422cb38e1e6bb20c5a60 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.150727 25875 leader_election.cc:302] T 681fdba8b3b2422cb38e1e6bb20c5a60 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bb61bde5c6d4e73a41f0f7a1e7777c0; no voters:
I20260808 07:03:29.150794 25875 leader_election.cc:288] T 681fdba8b3b2422cb38e1e6bb20c5a60 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:03:29.150807 25940 raft_consensus.cc:2781] T 681fdba8b3b2422cb38e1e6bb20c5a60 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:29.150869 25940 raft_consensus.cc:686] T 681fdba8b3b2422cb38e1e6bb20c5a60 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 LEADER]: Becoming Leader. State: Replica: 1bb61bde5c6d4e73a41f0f7a1e7777c0, State: Running, Role: LEADER
I20260808 07:03:29.150919 25940 consensus_queue.cc:237] T 681fdba8b3b2422cb38e1e6bb20c5a60 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.151321 25725 catalog_manager.cc:5495] T 681fdba8b3b2422cb38e1e6bb20c5a60 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bb61bde5c6d4e73a41f0f7a1e7777c0 (127.15.250.129). New cstate: current_term: 1 leader_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } health_report { overall_health: HEALTHY } } }
I20260808 07:03:29.152736 25849 tablet_service.cc:1514] Processing DeleteTablet for tablet 61c600f41d1e4e158c5e9f112bb448f2 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:38226
I20260808 07:03:29.152971 25966 tablet_replica.cc:330] T 61c600f41d1e4e158c5e9f112bb448f2 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: stopping tablet replica
I20260808 07:03:29.153132 25966 raft_consensus.cc:2229] T 61c600f41d1e4e158c5e9f112bb448f2 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 LEADER]: Raft consensus shutting down.
I20260808 07:03:29.153213 25966 raft_consensus.cc:2258] T 61c600f41d1e4e158c5e9f112bb448f2 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 07:03:29.153702 25966 ts_tablet_manager.cc:1905] T 61c600f41d1e4e158c5e9f112bb448f2 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 07:03:29.155141 25966 ts_tablet_manager.cc:1918] T 61c600f41d1e4e158c5e9f112bb448f2 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260808 07:03:29.155203 25966 log.cc:1196] T 61c600f41d1e4e158c5e9f112bb448f2 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Deleting WAL directory at /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/wals/61c600f41d1e4e158c5e9f112bb448f2
I20260808 07:03:29.156998 25849 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f629ac47ce046f5b650998fcb743ad0 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:38226
I20260808 07:03:29.157143 25966 tablet_replica.cc:330] T 9f629ac47ce046f5b650998fcb743ad0 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: stopping tablet replica
I20260808 07:03:29.157234 25966 raft_consensus.cc:2229] T 9f629ac47ce046f5b650998fcb743ad0 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 LEADER]: Raft consensus shutting down.
I20260808 07:03:29.157281 25966 raft_consensus.cc:2258] T 9f629ac47ce046f5b650998fcb743ad0 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 07:03:29.157521 25966 ts_tablet_manager.cc:1905] T 9f629ac47ce046f5b650998fcb743ad0 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 07:03:29.158747 25966 ts_tablet_manager.cc:1918] T 9f629ac47ce046f5b650998fcb743ad0 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260808 07:03:29.158812 25966 log.cc:1196] T 9f629ac47ce046f5b650998fcb743ad0 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Deleting WAL directory at /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/wals/9f629ac47ce046f5b650998fcb743ad0
I20260808 07:03:29.160249 25849 tablet_service.cc:1514] Processing DeleteTablet for tablet de6e1931655b4f049fcfffa9039de87c with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:38226
I20260808 07:03:29.160444 25966 tablet_replica.cc:330] T de6e1931655b4f049fcfffa9039de87c P 1bb61bde5c6d4e73a41f0f7a1e7777c0: stopping tablet replica
I20260808 07:03:29.160513 25966 raft_consensus.cc:2229] T de6e1931655b4f049fcfffa9039de87c P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 LEADER]: Raft consensus shutting down.
I20260808 07:03:29.160568 25966 raft_consensus.cc:2258] T de6e1931655b4f049fcfffa9039de87c P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 07:03:29.160842 25966 ts_tablet_manager.cc:1905] T de6e1931655b4f049fcfffa9039de87c P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 07:03:29.161906 25966 ts_tablet_manager.cc:1918] T de6e1931655b4f049fcfffa9039de87c P 1bb61bde5c6d4e73a41f0f7a1e7777c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260808 07:03:29.161955 25966 log.cc:1196] T de6e1931655b4f049fcfffa9039de87c P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Deleting WAL directory at /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/wals/de6e1931655b4f049fcfffa9039de87c
I20260808 07:03:29.163398 25849 tablet_service.cc:1514] Processing DeleteTablet for tablet 6922517948894072b8bec2b418cc0bb5 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:38226
I20260808 07:03:29.163524 25966 tablet_replica.cc:330] T 6922517948894072b8bec2b418cc0bb5 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: stopping tablet replica
I20260808 07:03:29.163586 25966 raft_consensus.cc:2229] T 6922517948894072b8bec2b418cc0bb5 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 LEADER]: Raft consensus shutting down.
I20260808 07:03:29.163640 25966 raft_consensus.cc:2258] T 6922517948894072b8bec2b418cc0bb5 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 07:03:29.163882 25966 ts_tablet_manager.cc:1905] T 6922517948894072b8bec2b418cc0bb5 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 07:03:29.165032 25966 ts_tablet_manager.cc:1918] T 6922517948894072b8bec2b418cc0bb5 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260808 07:03:29.165081 25966 log.cc:1196] T 6922517948894072b8bec2b418cc0bb5 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Deleting WAL directory at /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/wals/6922517948894072b8bec2b418cc0bb5
I20260808 07:03:29.166843 25849 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ad895cdfa404ab5bac618ca14edde34 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:38226
I20260808 07:03:29.166975 25966 tablet_replica.cc:330] T 0ad895cdfa404ab5bac618ca14edde34 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: stopping tablet replica
I20260808 07:03:29.167050 25966 raft_consensus.cc:2229] T 0ad895cdfa404ab5bac618ca14edde34 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 LEADER]: Raft consensus shutting down.
I20260808 07:03:29.167102 25966 raft_consensus.cc:2258] T 0ad895cdfa404ab5bac618ca14edde34 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 07:03:29.167438 25966 ts_tablet_manager.cc:1905] T 0ad895cdfa404ab5bac618ca14edde34 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 07:03:29.168591 25966 ts_tablet_manager.cc:1918] T 0ad895cdfa404ab5bac618ca14edde34 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260808 07:03:29.168660 25966 log.cc:1196] T 0ad895cdfa404ab5bac618ca14edde34 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Deleting WAL directory at /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/wals/0ad895cdfa404ab5bac618ca14edde34
I20260808 07:03:29.170145 25875 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "47c5d39ba24e45e0a148e9c8dc92a10d"
dest_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0"
from {username='slave'} at 127.0.0.1:38226
I20260808 07:03:29.170223 25875 raft_consensus.cc:580] T 47c5d39ba24e45e0a148e9c8dc92a10d P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 LEADER]: Received request to step down
I20260808 07:03:29.170240 25875 raft_consensus.cc:3032] T 47c5d39ba24e45e0a148e9c8dc92a10d P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 07:03:29.170258 25875 raft_consensus.cc:729] T 47c5d39ba24e45e0a148e9c8dc92a10d P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1bb61bde5c6d4e73a41f0f7a1e7777c0, State: Running, Role: LEADER
I20260808 07:03:29.170285 25875 consensus_queue.cc:260] T 47c5d39ba24e45e0a148e9c8dc92a10d P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.170324 25875 raft_consensus.cc:3037] T 47c5d39ba24e45e0a148e9c8dc92a10d P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 07:03:29.171622 25849 tablet_service.cc:1514] Processing DeleteTablet for tablet 47c5d39ba24e45e0a148e9c8dc92a10d with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:38226
I20260808 07:03:29.171725 25966 tablet_replica.cc:330] T 47c5d39ba24e45e0a148e9c8dc92a10d P 1bb61bde5c6d4e73a41f0f7a1e7777c0: stopping tablet replica
I20260808 07:03:29.171815 25966 raft_consensus.cc:2229] T 47c5d39ba24e45e0a148e9c8dc92a10d P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 07:03:29.171873 25966 raft_consensus.cc:2258] T 47c5d39ba24e45e0a148e9c8dc92a10d P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 07:03:29.172147 25966 ts_tablet_manager.cc:1905] T 47c5d39ba24e45e0a148e9c8dc92a10d P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 07:03:29.173637 25966 ts_tablet_manager.cc:1918] T 47c5d39ba24e45e0a148e9c8dc92a10d P 1bb61bde5c6d4e73a41f0f7a1e7777c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260808 07:03:29.173722 25966 log.cc:1196] T 47c5d39ba24e45e0a148e9c8dc92a10d P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Deleting WAL directory at /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/wals/47c5d39ba24e45e0a148e9c8dc92a10d
I20260808 07:03:29.175202 25875 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f24fedb649434547bc1d2f58f50bc963"
dest_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0"
from {username='slave'} at 127.0.0.1:38226
I20260808 07:03:29.175300 25875 raft_consensus.cc:580] T f24fedb649434547bc1d2f58f50bc963 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 LEADER]: Received request to step down
I20260808 07:03:29.175324 25875 raft_consensus.cc:3032] T f24fedb649434547bc1d2f58f50bc963 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 07:03:29.175338 25875 raft_consensus.cc:729] T f24fedb649434547bc1d2f58f50bc963 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1bb61bde5c6d4e73a41f0f7a1e7777c0, State: Running, Role: LEADER
I20260808 07:03:29.175395 25875 consensus_queue.cc:260] T f24fedb649434547bc1d2f58f50bc963 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.175451 25875 raft_consensus.cc:3037] T f24fedb649434547bc1d2f58f50bc963 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 07:03:29.176654 25849 tablet_service.cc:1514] Processing DeleteTablet for tablet f24fedb649434547bc1d2f58f50bc963 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:38226
I20260808 07:03:29.176753 25966 tablet_replica.cc:330] T f24fedb649434547bc1d2f58f50bc963 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: stopping tablet replica
I20260808 07:03:29.176873 25966 raft_consensus.cc:2229] T f24fedb649434547bc1d2f58f50bc963 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 07:03:29.176918 25966 raft_consensus.cc:2258] T f24fedb649434547bc1d2f58f50bc963 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 07:03:29.177191 25966 ts_tablet_manager.cc:1905] T f24fedb649434547bc1d2f58f50bc963 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 07:03:29.178629 25966 ts_tablet_manager.cc:1918] T f24fedb649434547bc1d2f58f50bc963 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260808 07:03:29.178725 25966 log.cc:1196] T f24fedb649434547bc1d2f58f50bc963 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Deleting WAL directory at /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/wals/f24fedb649434547bc1d2f58f50bc963
I20260808 07:03:29.180236 25875 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "619e3f62e1cc462dae57b054f2af0236"
dest_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0"
from {username='slave'} at 127.0.0.1:38226
I20260808 07:03:29.180354 25875 raft_consensus.cc:580] T 619e3f62e1cc462dae57b054f2af0236 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 LEADER]: Received request to step down
I20260808 07:03:29.180395 25875 raft_consensus.cc:3032] T 619e3f62e1cc462dae57b054f2af0236 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 07:03:29.180421 25875 raft_consensus.cc:729] T 619e3f62e1cc462dae57b054f2af0236 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1bb61bde5c6d4e73a41f0f7a1e7777c0, State: Running, Role: LEADER
I20260808 07:03:29.180465 25875 consensus_queue.cc:260] T 619e3f62e1cc462dae57b054f2af0236 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.180516 25875 raft_consensus.cc:3037] T 619e3f62e1cc462dae57b054f2af0236 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 07:03:29.181707 25849 tablet_service.cc:1514] Processing DeleteTablet for tablet 619e3f62e1cc462dae57b054f2af0236 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:38226
I20260808 07:03:29.181818 25966 tablet_replica.cc:330] T 619e3f62e1cc462dae57b054f2af0236 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: stopping tablet replica
I20260808 07:03:29.181881 25966 raft_consensus.cc:2229] T 619e3f62e1cc462dae57b054f2af0236 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 07:03:29.181931 25966 raft_consensus.cc:2258] T 619e3f62e1cc462dae57b054f2af0236 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 07:03:29.182232 25966 ts_tablet_manager.cc:1905] T 619e3f62e1cc462dae57b054f2af0236 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 07:03:29.183629 25966 ts_tablet_manager.cc:1918] T 619e3f62e1cc462dae57b054f2af0236 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260808 07:03:29.183694 25966 log.cc:1196] T 619e3f62e1cc462dae57b054f2af0236 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Deleting WAL directory at /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/wals/619e3f62e1cc462dae57b054f2af0236
I20260808 07:03:29.185163 25875 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1b76850c4e9349f0b1c95093f15d21c1"
dest_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0"
from {username='slave'} at 127.0.0.1:38226
I20260808 07:03:29.185231 25875 raft_consensus.cc:580] T 1b76850c4e9349f0b1c95093f15d21c1 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 LEADER]: Received request to step down
I20260808 07:03:29.185248 25875 raft_consensus.cc:3032] T 1b76850c4e9349f0b1c95093f15d21c1 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 07:03:29.185297 25875 raft_consensus.cc:729] T 1b76850c4e9349f0b1c95093f15d21c1 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1bb61bde5c6d4e73a41f0f7a1e7777c0, State: Running, Role: LEADER
I20260808 07:03:29.185349 25875 consensus_queue.cc:260] T 1b76850c4e9349f0b1c95093f15d21c1 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.185403 25875 raft_consensus.cc:3037] T 1b76850c4e9349f0b1c95093f15d21c1 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 07:03:29.186537 25849 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b76850c4e9349f0b1c95093f15d21c1 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:38226
I20260808 07:03:29.186631 25966 tablet_replica.cc:330] T 1b76850c4e9349f0b1c95093f15d21c1 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: stopping tablet replica
I20260808 07:03:29.186697 25966 raft_consensus.cc:2229] T 1b76850c4e9349f0b1c95093f15d21c1 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 07:03:29.186740 25966 raft_consensus.cc:2258] T 1b76850c4e9349f0b1c95093f15d21c1 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 07:03:29.186991 25966 ts_tablet_manager.cc:1905] T 1b76850c4e9349f0b1c95093f15d21c1 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 07:03:29.188189 25966 ts_tablet_manager.cc:1918] T 1b76850c4e9349f0b1c95093f15d21c1 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260808 07:03:29.188253 25966 log.cc:1196] T 1b76850c4e9349f0b1c95093f15d21c1 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Deleting WAL directory at /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/wals/1b76850c4e9349f0b1c95093f15d21c1
I20260808 07:03:29.189551 25875 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "681fdba8b3b2422cb38e1e6bb20c5a60"
dest_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0"
from {username='slave'} at 127.0.0.1:38226
I20260808 07:03:29.189615 25875 raft_consensus.cc:580] T 681fdba8b3b2422cb38e1e6bb20c5a60 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 LEADER]: Received request to step down
I20260808 07:03:29.189653 25875 raft_consensus.cc:3032] T 681fdba8b3b2422cb38e1e6bb20c5a60 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 07:03:29.189677 25875 raft_consensus.cc:729] T 681fdba8b3b2422cb38e1e6bb20c5a60 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1bb61bde5c6d4e73a41f0f7a1e7777c0, State: Running, Role: LEADER
I20260808 07:03:29.189718 25875 consensus_queue.cc:260] T 681fdba8b3b2422cb38e1e6bb20c5a60 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb61bde5c6d4e73a41f0f7a1e7777c0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 45633 } }
I20260808 07:03:29.189774 25875 raft_consensus.cc:3037] T 681fdba8b3b2422cb38e1e6bb20c5a60 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 07:03:29.190842 25849 tablet_service.cc:1514] Processing DeleteTablet for tablet 681fdba8b3b2422cb38e1e6bb20c5a60 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:38226
I20260808 07:03:29.190955 25966 tablet_replica.cc:330] T 681fdba8b3b2422cb38e1e6bb20c5a60 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: stopping tablet replica
I20260808 07:03:29.191018 25966 raft_consensus.cc:2229] T 681fdba8b3b2422cb38e1e6bb20c5a60 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 07:03:29.191057 25966 raft_consensus.cc:2258] T 681fdba8b3b2422cb38e1e6bb20c5a60 P 1bb61bde5c6d4e73a41f0f7a1e7777c0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 07:03:29.191313 25966 ts_tablet_manager.cc:1905] T 681fdba8b3b2422cb38e1e6bb20c5a60 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 07:03:29.192617 25966 ts_tablet_manager.cc:1918] T 681fdba8b3b2422cb38e1e6bb20c5a60 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260808 07:03:29.192693 25966 log.cc:1196] T 681fdba8b3b2422cb38e1e6bb20c5a60 P 1bb61bde5c6d4e73a41f0f7a1e7777c0: Deleting WAL directory at /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/wals/681fdba8b3b2422cb38e1e6bb20c5a60
I20260808 07:03:29.194738 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 25788
I20260808 07:03:29.202044 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 25693
2026-08-08T07:03:29Z chronyd exiting
[ OK ] NumReplicas/RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric/0 (467 ms)
[ RUN ] NumReplicas/RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric/1
I20260808 07:03:29.222316 16362 test_util.cc:268] Using random seed: -1209968988
I20260808 07:03:29.222827 16362 ts_itest-base.cc:112] Starting cluster with:
I20260808 07:03:29.222869 16362 ts_itest-base.cc:113] --------------
I20260808 07:03:29.222880 16362 ts_itest-base.cc:114] 3 tablet servers
I20260808 07:03:29.222891 16362 ts_itest-base.cc:115] 3 replicas per TS
I20260808 07:03:29.222901 16362 ts_itest-base.cc:116] --------------
2026-08-08T07:03:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:03:29Z Disabled control of system clock
I20260808 07:03:29.232026 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.250.190:38139
--webserver_interface=127.15.250.190
--webserver_port=0
--builtin_ntp_servers=127.15.250.148:42597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.250.190:38139
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20260808 07:03:29.313570 25975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:29.313795 25975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:29.313815 25975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:29.314927 25975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:03:29.314965 25975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:29.314977 25975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:03:29.314990 25975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:03:29.316322 25975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:42597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.15.250.190:38139
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.250.190:38139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.250.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.25975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:29.316519 25975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:29.316758 25975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:29.321606 25983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:29.321611 25980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:29.321627 25981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:29.322094 25975 server_base.cc:1034] running on GCE node
I20260808 07:03:29.322348 25975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:29.322649 25975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:29.323828 25975 hybrid_clock.cc:648] HybridClock initialized: now 1786172609323808 us; error 37 us; skew 500 ppm
I20260808 07:03:29.325165 25975 webserver.cc:460] Webserver started at http://127.15.250.190:33665/ using document root <none> and password file <none>
I20260808 07:03:29.325428 25975 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:29.325476 25975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:29.325639 25975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:29.326725 25975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "e8905d78b2184ee0bbb62b1fa23bf86b"
format_stamp: "Formatted at 2026-08-08 07:03:29 on dist-test-slave-0b58"
I20260808 07:03:29.327099 25975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "e8905d78b2184ee0bbb62b1fa23bf86b"
format_stamp: "Formatted at 2026-08-08 07:03:29 on dist-test-slave-0b58"
I20260808 07:03:29.328483 25975 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 07:03:29.329213 25990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.329384 25975 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:03:29.329491 25975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e8905d78b2184ee0bbb62b1fa23bf86b"
format_stamp: "Formatted at 2026-08-08 07:03:29 on dist-test-slave-0b58"
I20260808 07:03:29.329562 25975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:29.343928 25975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:29.344378 25975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:29.344547 25975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:29.348786 25975 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.190:38139
I20260808 07:03:29.348840 26042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.190:38139 every 8 connection(s)
I20260808 07:03:29.349179 25975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260808 07:03:29.349843 26043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.352447 26043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8905d78b2184ee0bbb62b1fa23bf86b: Bootstrap starting.
I20260808 07:03:29.353014 26043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8905d78b2184ee0bbb62b1fa23bf86b: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.353255 26043 log.cc:824] T 00000000000000000000000000000000 P e8905d78b2184ee0bbb62b1fa23bf86b: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:29.353864 26043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8905d78b2184ee0bbb62b1fa23bf86b: No bootstrap required, opened a new log
I20260808 07:03:29.355377 26043 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8905d78b2184ee0bbb62b1fa23bf86b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8905d78b2184ee0bbb62b1fa23bf86b" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 38139 } }
I20260808 07:03:29.355515 26043 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8905d78b2184ee0bbb62b1fa23bf86b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:03:29.355537 26043 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8905d78b2184ee0bbb62b1fa23bf86b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8905d78b2184ee0bbb62b1fa23bf86b, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.355669 26043 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8905d78b2184ee0bbb62b1fa23bf86b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8905d78b2184ee0bbb62b1fa23bf86b" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 38139 } }
I20260808 07:03:29.355760 26043 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8905d78b2184ee0bbb62b1fa23bf86b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:03:29.355783 26043 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8905d78b2184ee0bbb62b1fa23bf86b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:03:29.355813 26043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8905d78b2184ee0bbb62b1fa23bf86b [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.356359 26043 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8905d78b2184ee0bbb62b1fa23bf86b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8905d78b2184ee0bbb62b1fa23bf86b" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 38139 } }
I20260808 07:03:29.356448 26043 leader_election.cc:302] T 00000000000000000000000000000000 P e8905d78b2184ee0bbb62b1fa23bf86b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8905d78b2184ee0bbb62b1fa23bf86b; no voters:
I20260808 07:03:29.356568 26043 leader_election.cc:288] T 00000000000000000000000000000000 P e8905d78b2184ee0bbb62b1fa23bf86b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:03:29.356667 26046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8905d78b2184ee0bbb62b1fa23bf86b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:29.356783 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 25975
I20260808 07:03:29.356848 26043 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8905d78b2184ee0bbb62b1fa23bf86b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:03:29.356887 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260808 07:03:29.356863 26046 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8905d78b2184ee0bbb62b1fa23bf86b [term 1 LEADER]: Becoming Leader. State: Replica: e8905d78b2184ee0bbb62b1fa23bf86b, State: Running, Role: LEADER
I20260808 07:03:29.357009 26046 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8905d78b2184ee0bbb62b1fa23bf86b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8905d78b2184ee0bbb62b1fa23bf86b" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 38139 } }
I20260808 07:03:29.357393 26046 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8905d78b2184ee0bbb62b1fa23bf86b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8905d78b2184ee0bbb62b1fa23bf86b. Latest consensus state: current_term: 1 leader_uuid: "e8905d78b2184ee0bbb62b1fa23bf86b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8905d78b2184ee0bbb62b1fa23bf86b" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 38139 } } }
I20260808 07:03:29.357508 26046 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8905d78b2184ee0bbb62b1fa23bf86b [sys.catalog]: This master's current role is: LEADER
I20260808 07:03:29.357394 26047 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8905d78b2184ee0bbb62b1fa23bf86b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8905d78b2184ee0bbb62b1fa23bf86b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8905d78b2184ee0bbb62b1fa23bf86b" member_type: VOTER last_known_addr { host: "127.15.250.190" port: 38139 } } }
I20260808 07:03:29.358104 26047 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8905d78b2184ee0bbb62b1fa23bf86b [sys.catalog]: This master's current role is: LEADER
I20260808 07:03:29.358229 26055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:03:29.358872 26055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:03:29.360445 26055 catalog_manager.cc:1347] Generated new cluster ID: 147df3d6086b458e8cbc3fdd1a50d926
I20260808 07:03:29.360497 26055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:03:29.378806 26055 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:03:29.379565 26055 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:03:29.384127 26055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8905d78b2184ee0bbb62b1fa23bf86b: Generated new TSK 0
I20260808 07:03:29.384352 26055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:03:29.387666 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.129:0
--local_ip_for_outbound_sockets=127.15.250.129
--webserver_interface=127.15.250.129
--webserver_port=0
--tserver_master_addrs=127.15.250.190:38139
--builtin_ntp_servers=127.15.250.148:42597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20260808 07:03:29.492089 26067 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 07:03:29.492373 26067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:29.492396 26067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:29.492432 26067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:29.493639 26067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:29.493692 26067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.129
I20260808 07:03:29.495056 26067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:42597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.250.129
--webserver_port=0
--tserver_master_addrs=127.15.250.190:38139
--never_fsync=true
--heap_profile_path=/tmp/kudu.26067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.129
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:29.495318 26067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:29.495630 26067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:29.501722 26073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:29.501847 26075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:29.501756 26067 server_base.cc:1034] running on GCE node
W20260808 07:03:29.501713 26072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:29.502126 26067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:29.502380 26067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:29.503547 26067 hybrid_clock.cc:648] HybridClock initialized: now 1786172609503535 us; error 30 us; skew 500 ppm
I20260808 07:03:29.504864 26067 webserver.cc:460] Webserver started at http://127.15.250.129:40855/ using document root <none> and password file <none>
I20260808 07:03:29.505115 26067 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:29.505184 26067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:29.505331 26067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:29.506338 26067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "9db3325ac3e244919f866a2e04b9b9a0"
format_stamp: "Formatted at 2026-08-08 07:03:29 on dist-test-slave-0b58"
I20260808 07:03:29.506733 26067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "9db3325ac3e244919f866a2e04b9b9a0"
format_stamp: "Formatted at 2026-08-08 07:03:29 on dist-test-slave-0b58"
I20260808 07:03:29.508148 26067 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 07:03:29.509020 26081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.509236 26067 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 07:03:29.509377 26067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "9db3325ac3e244919f866a2e04b9b9a0"
format_stamp: "Formatted at 2026-08-08 07:03:29 on dist-test-slave-0b58"
I20260808 07:03:29.509460 26067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:29.523366 26067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:29.523653 26067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:29.523757 26067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:29.523943 26067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:03:29.524350 26067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:03:29.524391 26067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.524430 26067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:03:29.524456 26067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.530599 26067 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.129:42123
I20260808 07:03:29.530678 26197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.129:42123 every 8 connection(s)
I20260808 07:03:29.531013 26067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260808 07:03:29.534559 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 26067
I20260808 07:03:29.534669 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260808 07:03:29.535888 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.130:0
--local_ip_for_outbound_sockets=127.15.250.130
--webserver_interface=127.15.250.130
--webserver_port=0
--tserver_master_addrs=127.15.250.190:38139
--builtin_ntp_servers=127.15.250.148:42597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260808 07:03:29.538278 26198 heartbeater.cc:347] Connected to a master server at 127.15.250.190:38139
I20260808 07:03:29.538403 26198 heartbeater.cc:464] Registering TS with master...
I20260808 07:03:29.538618 26198 heartbeater.cc:511] Master 127.15.250.190:38139 requested a full tablet report, sending...
I20260808 07:03:29.539175 26007 ts_manager.cc:194] Registered new tserver with Master: 9db3325ac3e244919f866a2e04b9b9a0 (127.15.250.129:42123)
I20260808 07:03:29.539830 26007 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.129:40879
W20260808 07:03:29.622085 26201 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 07:03:29.622301 26201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:29.622318 26201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:29.622336 26201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:29.623638 26201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:29.623701 26201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.130
I20260808 07:03:29.625126 26201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:42597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.250.130
--webserver_port=0
--tserver_master_addrs=127.15.250.190:38139
--never_fsync=true
--heap_profile_path=/tmp/kudu.26201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.130
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:29.625392 26201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:29.625706 26201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:03:29.631399 26207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:29.631385 26208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:29.631385 26210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:29.631578 26201 server_base.cc:1034] running on GCE node
I20260808 07:03:29.631817 26201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:29.632086 26201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:29.633260 26201 hybrid_clock.cc:648] HybridClock initialized: now 1786172609633238 us; error 35 us; skew 500 ppm
I20260808 07:03:29.634581 26201 webserver.cc:460] Webserver started at http://127.15.250.130:39879/ using document root <none> and password file <none>
I20260808 07:03:29.634846 26201 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:29.634895 26201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:29.635020 26201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:29.636029 26201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "117e71d831de49c89d100e8d430e6f91"
format_stamp: "Formatted at 2026-08-08 07:03:29 on dist-test-slave-0b58"
I20260808 07:03:29.636456 26201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "117e71d831de49c89d100e8d430e6f91"
format_stamp: "Formatted at 2026-08-08 07:03:29 on dist-test-slave-0b58"
I20260808 07:03:29.637873 26201 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 07:03:29.638571 26216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.638738 26201 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 07:03:29.638818 26201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "117e71d831de49c89d100e8d430e6f91"
format_stamp: "Formatted at 2026-08-08 07:03:29 on dist-test-slave-0b58"
I20260808 07:03:29.638899 26201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:29.652305 26201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:29.652623 26201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:29.652779 26201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:29.652995 26201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:03:29.653343 26201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:03:29.653396 26201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.653437 26201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:03:29.653465 26201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.659396 26201 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.130:36973
I20260808 07:03:29.659471 26331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.130:36973 every 8 connection(s)
I20260808 07:03:29.659814 26201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260808 07:03:29.662599 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 26201
I20260808 07:03:29.662701 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260808 07:03:29.663812 16362 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDkEqNX/build/release/bin/kudu
/tmp/dist-test-taskDkEqNX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.250.131:0
--local_ip_for_outbound_sockets=127.15.250.131
--webserver_interface=127.15.250.131
--webserver_port=0
--tserver_master_addrs=127.15.250.190:38139
--builtin_ntp_servers=127.15.250.148:42597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260808 07:03:29.666062 26332 heartbeater.cc:347] Connected to a master server at 127.15.250.190:38139
I20260808 07:03:29.666191 26332 heartbeater.cc:464] Registering TS with master...
I20260808 07:03:29.666401 26332 heartbeater.cc:511] Master 127.15.250.190:38139 requested a full tablet report, sending...
I20260808 07:03:29.666795 26007 ts_manager.cc:194] Registered new tserver with Master: 117e71d831de49c89d100e8d430e6f91 (127.15.250.130:36973)
I20260808 07:03:29.667176 26007 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.130:45799
W20260808 07:03:29.748540 26335 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 07:03:29.748762 26335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:03:29.748780 26335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:03:29.748798 26335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:03:29.749922 26335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:03:29.749974 26335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.250.131
I20260808 07:03:29.751267 26335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.250.148:42597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.250.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.250.131
--webserver_port=0
--tserver_master_addrs=127.15.250.190:38139
--never_fsync=true
--heap_profile_path=/tmp/kudu.26335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.250.131
--log_dir=/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:42:04 UTC on ccycloud.quasar-fxrzxs.root.comops.site
build id 4574018
I20260808 07:03:29.751468 26335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:03:29.751735 26335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 07:03:29.756844 26335 server_base.cc:1034] running on GCE node
W20260808 07:03:29.756827 26343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:29.756860 26341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:03:29.756827 26340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:03:29.757305 26335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:03:29.757552 26335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:03:29.758715 26335 hybrid_clock.cc:648] HybridClock initialized: now 1786172609758703 us; error 42 us; skew 500 ppm
I20260808 07:03:29.759970 26335 webserver.cc:460] Webserver started at http://127.15.250.131:39867/ using document root <none> and password file <none>
I20260808 07:03:29.760226 26335 fs_manager.cc:359] Metadata directory not provided
I20260808 07:03:29.760337 26335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:03:29.760488 26335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:03:29.761538 26335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b4106f2b661f4faaa255d5636a29788a"
format_stamp: "Formatted at 2026-08-08 07:03:29 on dist-test-slave-0b58"
I20260808 07:03:29.761896 26335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b4106f2b661f4faaa255d5636a29788a"
format_stamp: "Formatted at 2026-08-08 07:03:29 on dist-test-slave-0b58"
I20260808 07:03:29.763295 26335 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 07:03:29.764030 26349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.764220 26335 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 07:03:29.764356 26335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b4106f2b661f4faaa255d5636a29788a"
format_stamp: "Formatted at 2026-08-08 07:03:29 on dist-test-slave-0b58"
I20260808 07:03:29.764416 26335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:03:29.783933 26335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:03:29.784226 26335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:03:29.784518 26335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:03:29.784708 26335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:03:29.785017 26335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:03:29.785048 26335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.785071 26335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:03:29.785084 26335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.790934 26335 rpc_server.cc:307] RPC server started. Bound to: 127.15.250.131:34037
I20260808 07:03:29.791015 26465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.250.131:34037 every 8 connection(s)
I20260808 07:03:29.791363 26335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260808 07:03:29.796778 26466 heartbeater.cc:347] Connected to a master server at 127.15.250.190:38139
I20260808 07:03:29.796895 26466 heartbeater.cc:464] Registering TS with master...
I20260808 07:03:29.797075 26466 heartbeater.cc:511] Master 127.15.250.190:38139 requested a full tablet report, sending...
I20260808 07:03:29.797451 26007 ts_manager.cc:194] Registered new tserver with Master: b4106f2b661f4faaa255d5636a29788a (127.15.250.131:34037)
I20260808 07:03:29.797829 26007 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.250.131:57347
I20260808 07:03:29.800202 16362 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDkEqNX/build/release/bin/kudu as pid 26335
I20260808 07:03:29.800343 16362 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260808 07:03:29.801438 16362 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 07:03:29.808621 16362 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 07:03:29.808677 16362 test_util.cc:268] Using random seed: -1209382619
I20260808 07:03:29.815275 26007 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40496:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 07:03:29.815699 26007 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 07:03:29.823004 26265 tablet_service.cc:1467] Processing CreateTablet for tablet dad0a14869924e4a895536588073b33b (DEFAULT_TABLE table=test-workload [id=bd244df9d0d14a4fa0c29579cec9d0a3]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260808 07:03:29.823082 26264 tablet_service.cc:1467] Processing CreateTablet for tablet c3f1db05fe244bae87a077fba49f764d (DEFAULT_TABLE table=test-workload [id=bd244df9d0d14a4fa0c29579cec9d0a3]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260808 07:03:29.823326 26263 tablet_service.cc:1467] Processing CreateTablet for tablet 97901242097c446ebf1d046fe86d0800 (DEFAULT_TABLE table=test-workload [id=bd244df9d0d14a4fa0c29579cec9d0a3]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260808 07:03:29.823010 26257 tablet_service.cc:1467] Processing CreateTablet for tablet 0080f8a2994e4252802ea6c5a6f4863b (DEFAULT_TABLE table=test-workload [id=bd244df9d0d14a4fa0c29579cec9d0a3]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260808 07:03:29.823423 26262 tablet_service.cc:1467] Processing CreateTablet for tablet e0d8f0133ef04fbcb07c3e93ce4a8382 (DEFAULT_TABLE table=test-workload [id=bd244df9d0d14a4fa0c29579cec9d0a3]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260808 07:03:29.823503 26261 tablet_service.cc:1467] Processing CreateTablet for tablet 55dd567790424028b86b3883a524be4a (DEFAULT_TABLE table=test-workload [id=bd244df9d0d14a4fa0c29579cec9d0a3]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260808 07:03:29.823596 26260 tablet_service.cc:1467] Processing CreateTablet for tablet 051057f218174b81bf33ba8a5834e9da (DEFAULT_TABLE table=test-workload [id=bd244df9d0d14a4fa0c29579cec9d0a3]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260808 07:03:29.823582 26265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dad0a14869924e4a895536588073b33b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.823782 26257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0080f8a2994e4252802ea6c5a6f4863b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.824111 26259 tablet_service.cc:1467] Processing CreateTablet for tablet f64d7ac8e8894a81bbbe7d9f364f9ea6 (DEFAULT_TABLE table=test-workload [id=bd244df9d0d14a4fa0c29579cec9d0a3]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260808 07:03:29.824263 26258 tablet_service.cc:1467] Processing CreateTablet for tablet 3f58a7559c834237966455a46e2c6d02 (DEFAULT_TABLE table=test-workload [id=bd244df9d0d14a4fa0c29579cec9d0a3]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260808 07:03:29.824409 26256 tablet_service.cc:1467] Processing CreateTablet for tablet 1733039f2bdd4c4fa9a47522cbdefe9e (DEFAULT_TABLE table=test-workload [id=bd244df9d0d14a4fa0c29579cec9d0a3]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260808 07:03:29.824374 26397 tablet_service.cc:1467] Processing CreateTablet for tablet 97901242097c446ebf1d046fe86d0800 (DEFAULT_TABLE table=test-workload [id=bd244df9d0d14a4fa0c29579cec9d0a3]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260808 07:03:29.824764 26397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97901242097c446ebf1d046fe86d0800. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.824977 26399 tablet_service.cc:1467] Processing CreateTablet for tablet dad0a14869924e4a895536588073b33b (DEFAULT_TABLE table=test-workload [id=bd244df9d0d14a4fa0c29579cec9d0a3]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260808 07:03:29.825088 26399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dad0a14869924e4a895536588073b33b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.825109 26131 tablet_service.cc:1467] Processing CreateTablet for tablet dad0a14869924e4a895536588073b33b (DEFAULT_TABLE table=test-workload [id=bd244df9d0d14a4fa0c29579cec9d0a3]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260808 07:03:29.825289 26398 tablet_service.cc:1467] Processing CreateTablet for tablet c3f1db05fe244bae87a077fba49f764d (DEFAULT_TABLE table=test-workload [id=bd244df9d0d14a4fa0c29579cec9d0a3]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260808 07:03:29.825371 26398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3f1db05fe244bae87a077fba49f764d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.825471 26131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dad0a14869924e4a895536588073b33b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.825608 26396 tablet_service.cc:1467] Processing CreateTablet for tablet e0d8f0133ef04fbcb07c3e93ce4a8382 (DEFAULT_TABLE table=test-workload [id=bd244df9d0d14a4fa0c29579cec9d0a3]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260808 07:03:29.825693 26396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0d8f0133ef04fbcb07c3e93ce4a8382. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.826957 26395 tablet_service.cc:1467] Processing CreateTablet for tablet 55dd567790424028b86b3883a524be4a (DEFAULT_TABLE table=test-workload [id=bd244df9d0d14a4fa0c29579cec9d0a3]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260808 07:03:29.827047 26395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55dd567790424028b86b3883a524be4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.828169 26485 tablet_bootstrap.cc:492] T 97901242097c446ebf1d046fe86d0800 P b4106f2b661f4faaa255d5636a29788a: Bootstrap starting.
I20260808 07:03:29.828320 26393 tablet_service.cc:1467] Processing CreateTablet for tablet 051057f218174b81bf33ba8a5834e9da (DEFAULT_TABLE table=test-workload [id=bd244df9d0d14a4fa0c29579cec9d0a3]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260808 07:03:29.828414 26393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 051057f218174b81bf33ba8a5834e9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.829272 26392 tablet_service.cc:1467] Processing CreateTablet for tablet f64d7ac8e8894a81bbbe7d9f364f9ea6 (DEFAULT_TABLE table=test-workload [id=bd244df9d0d14a4fa0c29579cec9d0a3]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260808 07:03:29.829291 26485 tablet_bootstrap.cc:654] T 97901242097c446ebf1d046fe86d0800 P b4106f2b661f4faaa255d5636a29788a: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.829380 26392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f64d7ac8e8894a81bbbe7d9f364f9ea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.829591 26485 log.cc:824] T 97901242097c446ebf1d046fe86d0800 P b4106f2b661f4faaa255d5636a29788a: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:29.830206 26391 tablet_service.cc:1467] Processing CreateTablet for tablet 3f58a7559c834237966455a46e2c6d02 (DEFAULT_TABLE table=test-workload [id=bd244df9d0d14a4fa0c29579cec9d0a3]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260808 07:03:29.829948 26486 tablet_bootstrap.cc:492] T dad0a14869924e4a895536588073b33b P 9db3325ac3e244919f866a2e04b9b9a0: Bootstrap starting.
I20260808 07:03:29.830278 26391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f58a7559c834237966455a46e2c6d02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.830295 26485 tablet_bootstrap.cc:492] T 97901242097c446ebf1d046fe86d0800 P b4106f2b661f4faaa255d5636a29788a: No bootstrap required, opened a new log
I20260808 07:03:29.830418 26485 ts_tablet_manager.cc:1397] T 97901242097c446ebf1d046fe86d0800 P b4106f2b661f4faaa255d5636a29788a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 07:03:29.830920 26486 tablet_bootstrap.cc:654] T dad0a14869924e4a895536588073b33b P 9db3325ac3e244919f866a2e04b9b9a0: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.831221 26486 log.cc:824] T dad0a14869924e4a895536588073b33b P 9db3325ac3e244919f866a2e04b9b9a0: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:29.831413 26390 tablet_service.cc:1467] Processing CreateTablet for tablet 0080f8a2994e4252802ea6c5a6f4863b (DEFAULT_TABLE table=test-workload [id=bd244df9d0d14a4fa0c29579cec9d0a3]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260808 07:03:29.831544 26390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0080f8a2994e4252802ea6c5a6f4863b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.831970 26486 tablet_bootstrap.cc:492] T dad0a14869924e4a895536588073b33b P 9db3325ac3e244919f866a2e04b9b9a0: No bootstrap required, opened a new log
I20260808 07:03:29.832078 26486 ts_tablet_manager.cc:1397] T dad0a14869924e4a895536588073b33b P 9db3325ac3e244919f866a2e04b9b9a0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 07:03:29.832479 26121 tablet_service.cc:1467] Processing CreateTablet for tablet 1733039f2bdd4c4fa9a47522cbdefe9e (DEFAULT_TABLE table=test-workload [id=bd244df9d0d14a4fa0c29579cec9d0a3]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260808 07:03:29.832607 26121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1733039f2bdd4c4fa9a47522cbdefe9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.833693 26123 tablet_service.cc:1467] Processing CreateTablet for tablet 0080f8a2994e4252802ea6c5a6f4863b (DEFAULT_TABLE table=test-workload [id=bd244df9d0d14a4fa0c29579cec9d0a3]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260808 07:03:29.833596 26485 raft_consensus.cc:348] T 97901242097c446ebf1d046fe86d0800 P b4106f2b661f4faaa255d5636a29788a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.833772 26485 raft_consensus.cc:729] T 97901242097c446ebf1d046fe86d0800 P b4106f2b661f4faaa255d5636a29788a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4106f2b661f4faaa255d5636a29788a, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.833806 26123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0080f8a2994e4252802ea6c5a6f4863b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.833860 26485 consensus_queue.cc:260] T 97901242097c446ebf1d046fe86d0800 P b4106f2b661f4faaa255d5636a29788a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.834105 26485 ts_tablet_manager.cc:1428] T 97901242097c446ebf1d046fe86d0800 P b4106f2b661f4faaa255d5636a29788a: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260808 07:03:29.834227 26485 tablet_bootstrap.cc:492] T 55dd567790424028b86b3883a524be4a P b4106f2b661f4faaa255d5636a29788a: Bootstrap starting.
I20260808 07:03:29.834246 26466 heartbeater.cc:503] Master 127.15.250.190:38139 was elected leader, sending a full tablet report...
I20260808 07:03:29.834600 26485 tablet_bootstrap.cc:654] T 55dd567790424028b86b3883a524be4a P b4106f2b661f4faaa255d5636a29788a: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.835017 26124 tablet_service.cc:1467] Processing CreateTablet for tablet 3f58a7559c834237966455a46e2c6d02 (DEFAULT_TABLE table=test-workload [id=bd244df9d0d14a4fa0c29579cec9d0a3]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260808 07:03:29.835141 26124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f58a7559c834237966455a46e2c6d02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.835211 26125 tablet_service.cc:1467] Processing CreateTablet for tablet f64d7ac8e8894a81bbbe7d9f364f9ea6 (DEFAULT_TABLE table=test-workload [id=bd244df9d0d14a4fa0c29579cec9d0a3]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260808 07:03:29.835232 26485 tablet_bootstrap.cc:492] T 55dd567790424028b86b3883a524be4a P b4106f2b661f4faaa255d5636a29788a: No bootstrap required, opened a new log
I20260808 07:03:29.835271 26485 ts_tablet_manager.cc:1397] T 55dd567790424028b86b3883a524be4a P b4106f2b661f4faaa255d5636a29788a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260808 07:03:29.835341 26125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f64d7ac8e8894a81bbbe7d9f364f9ea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.835424 26485 raft_consensus.cc:348] T 55dd567790424028b86b3883a524be4a P b4106f2b661f4faaa255d5636a29788a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.835501 26485 raft_consensus.cc:729] T 55dd567790424028b86b3883a524be4a P b4106f2b661f4faaa255d5636a29788a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4106f2b661f4faaa255d5636a29788a, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.835530 26485 consensus_queue.cc:260] T 55dd567790424028b86b3883a524be4a P b4106f2b661f4faaa255d5636a29788a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.835688 26485 ts_tablet_manager.cc:1428] T 55dd567790424028b86b3883a524be4a P b4106f2b661f4faaa255d5636a29788a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.835745 26485 tablet_bootstrap.cc:492] T 051057f218174b81bf33ba8a5834e9da P b4106f2b661f4faaa255d5636a29788a: Bootstrap starting.
I20260808 07:03:29.836138 26485 tablet_bootstrap.cc:654] T 051057f218174b81bf33ba8a5834e9da P b4106f2b661f4faaa255d5636a29788a: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.835125 26486 raft_consensus.cc:348] T dad0a14869924e4a895536588073b33b P 9db3325ac3e244919f866a2e04b9b9a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } } peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } }
I20260808 07:03:29.836453 26389 tablet_service.cc:1467] Processing CreateTablet for tablet 1733039f2bdd4c4fa9a47522cbdefe9e (DEFAULT_TABLE table=test-workload [id=bd244df9d0d14a4fa0c29579cec9d0a3]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260808 07:03:29.836496 26486 raft_consensus.cc:729] T dad0a14869924e4a895536588073b33b P 9db3325ac3e244919f866a2e04b9b9a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db3325ac3e244919f866a2e04b9b9a0, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.836568 26389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1733039f2bdd4c4fa9a47522cbdefe9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.836570 26486 consensus_queue.cc:260] T dad0a14869924e4a895536588073b33b P 9db3325ac3e244919f866a2e04b9b9a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } } peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } }
I20260808 07:03:29.836836 26486 ts_tablet_manager.cc:1428] T dad0a14869924e4a895536588073b33b P 9db3325ac3e244919f866a2e04b9b9a0: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260808 07:03:29.836886 26198 heartbeater.cc:503] Master 127.15.250.190:38139 was elected leader, sending a full tablet report...
I20260808 07:03:29.836917 26486 tablet_bootstrap.cc:492] T 1733039f2bdd4c4fa9a47522cbdefe9e P 9db3325ac3e244919f866a2e04b9b9a0: Bootstrap starting.
I20260808 07:03:29.837292 26486 tablet_bootstrap.cc:654] T 1733039f2bdd4c4fa9a47522cbdefe9e P 9db3325ac3e244919f866a2e04b9b9a0: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.837862 26486 tablet_bootstrap.cc:492] T 1733039f2bdd4c4fa9a47522cbdefe9e P 9db3325ac3e244919f866a2e04b9b9a0: No bootstrap required, opened a new log
I20260808 07:03:29.837904 26486 ts_tablet_manager.cc:1397] T 1733039f2bdd4c4fa9a47522cbdefe9e P 9db3325ac3e244919f866a2e04b9b9a0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 07:03:29.838068 26486 raft_consensus.cc:348] T 1733039f2bdd4c4fa9a47522cbdefe9e P 9db3325ac3e244919f866a2e04b9b9a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.838151 26486 raft_consensus.cc:729] T 1733039f2bdd4c4fa9a47522cbdefe9e P 9db3325ac3e244919f866a2e04b9b9a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db3325ac3e244919f866a2e04b9b9a0, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.838132 26129 tablet_service.cc:1467] Processing CreateTablet for tablet 97901242097c446ebf1d046fe86d0800 (DEFAULT_TABLE table=test-workload [id=bd244df9d0d14a4fa0c29579cec9d0a3]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260808 07:03:29.838289 26126 tablet_service.cc:1467] Processing CreateTablet for tablet 051057f218174b81bf33ba8a5834e9da (DEFAULT_TABLE table=test-workload [id=bd244df9d0d14a4fa0c29579cec9d0a3]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260808 07:03:29.838303 26129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97901242097c446ebf1d046fe86d0800. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.838239 26486 consensus_queue.cc:260] T 1733039f2bdd4c4fa9a47522cbdefe9e P 9db3325ac3e244919f866a2e04b9b9a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.838356 26485 tablet_bootstrap.cc:492] T 051057f218174b81bf33ba8a5834e9da P b4106f2b661f4faaa255d5636a29788a: No bootstrap required, opened a new log
I20260808 07:03:29.838394 26485 ts_tablet_manager.cc:1397] T 051057f218174b81bf33ba8a5834e9da P b4106f2b661f4faaa255d5636a29788a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260808 07:03:29.838403 26126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 051057f218174b81bf33ba8a5834e9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.838405 26486 ts_tablet_manager.cc:1428] T 1733039f2bdd4c4fa9a47522cbdefe9e P 9db3325ac3e244919f866a2e04b9b9a0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.838506 26486 tablet_bootstrap.cc:492] T 0080f8a2994e4252802ea6c5a6f4863b P 9db3325ac3e244919f866a2e04b9b9a0: Bootstrap starting.
I20260808 07:03:29.838537 26485 raft_consensus.cc:348] T 051057f218174b81bf33ba8a5834e9da P b4106f2b661f4faaa255d5636a29788a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.838599 26485 raft_consensus.cc:729] T 051057f218174b81bf33ba8a5834e9da P b4106f2b661f4faaa255d5636a29788a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4106f2b661f4faaa255d5636a29788a, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.838626 26485 consensus_queue.cc:260] T 051057f218174b81bf33ba8a5834e9da P b4106f2b661f4faaa255d5636a29788a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.838707 26485 ts_tablet_manager.cc:1428] T 051057f218174b81bf33ba8a5834e9da P b4106f2b661f4faaa255d5636a29788a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.838754 26485 tablet_bootstrap.cc:492] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P b4106f2b661f4faaa255d5636a29788a: Bootstrap starting.
I20260808 07:03:29.838899 26486 tablet_bootstrap.cc:654] T 0080f8a2994e4252802ea6c5a6f4863b P 9db3325ac3e244919f866a2e04b9b9a0: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.839102 26485 tablet_bootstrap.cc:654] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P b4106f2b661f4faaa255d5636a29788a: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.839823 26486 tablet_bootstrap.cc:492] T 0080f8a2994e4252802ea6c5a6f4863b P 9db3325ac3e244919f866a2e04b9b9a0: No bootstrap required, opened a new log
I20260808 07:03:29.839855 26486 ts_tablet_manager.cc:1397] T 0080f8a2994e4252802ea6c5a6f4863b P 9db3325ac3e244919f866a2e04b9b9a0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 07:03:29.839879 26485 tablet_bootstrap.cc:492] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P b4106f2b661f4faaa255d5636a29788a: No bootstrap required, opened a new log
I20260808 07:03:29.839936 26485 ts_tablet_manager.cc:1397] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P b4106f2b661f4faaa255d5636a29788a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260808 07:03:29.839999 26486 raft_consensus.cc:348] T 0080f8a2994e4252802ea6c5a6f4863b P 9db3325ac3e244919f866a2e04b9b9a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.840062 26486 raft_consensus.cc:729] T 0080f8a2994e4252802ea6c5a6f4863b P 9db3325ac3e244919f866a2e04b9b9a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db3325ac3e244919f866a2e04b9b9a0, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.840078 26485 raft_consensus.cc:348] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P b4106f2b661f4faaa255d5636a29788a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.840092 26486 consensus_queue.cc:260] T 0080f8a2994e4252802ea6c5a6f4863b P 9db3325ac3e244919f866a2e04b9b9a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.840137 26485 raft_consensus.cc:729] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P b4106f2b661f4faaa255d5636a29788a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4106f2b661f4faaa255d5636a29788a, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.840164 26485 consensus_queue.cc:260] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P b4106f2b661f4faaa255d5636a29788a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.840189 26486 ts_tablet_manager.cc:1428] T 0080f8a2994e4252802ea6c5a6f4863b P 9db3325ac3e244919f866a2e04b9b9a0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.840241 26486 tablet_bootstrap.cc:492] T 3f58a7559c834237966455a46e2c6d02 P 9db3325ac3e244919f866a2e04b9b9a0: Bootstrap starting.
I20260808 07:03:29.840281 26485 ts_tablet_manager.cc:1428] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P b4106f2b661f4faaa255d5636a29788a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.840332 26485 tablet_bootstrap.cc:492] T 3f58a7559c834237966455a46e2c6d02 P b4106f2b661f4faaa255d5636a29788a: Bootstrap starting.
I20260808 07:03:29.838177 26128 tablet_service.cc:1467] Processing CreateTablet for tablet e0d8f0133ef04fbcb07c3e93ce4a8382 (DEFAULT_TABLE table=test-workload [id=bd244df9d0d14a4fa0c29579cec9d0a3]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260808 07:03:29.840615 26486 tablet_bootstrap.cc:654] T 3f58a7559c834237966455a46e2c6d02 P 9db3325ac3e244919f866a2e04b9b9a0: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.840687 26485 tablet_bootstrap.cc:654] T 3f58a7559c834237966455a46e2c6d02 P b4106f2b661f4faaa255d5636a29788a: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.840724 26128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0d8f0133ef04fbcb07c3e93ce4a8382. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.841221 26486 tablet_bootstrap.cc:492] T 3f58a7559c834237966455a46e2c6d02 P 9db3325ac3e244919f866a2e04b9b9a0: No bootstrap required, opened a new log
I20260808 07:03:29.841264 26486 ts_tablet_manager.cc:1397] T 3f58a7559c834237966455a46e2c6d02 P 9db3325ac3e244919f866a2e04b9b9a0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 07:03:29.841395 26486 raft_consensus.cc:348] T 3f58a7559c834237966455a46e2c6d02 P 9db3325ac3e244919f866a2e04b9b9a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.841439 26486 raft_consensus.cc:729] T 3f58a7559c834237966455a46e2c6d02 P 9db3325ac3e244919f866a2e04b9b9a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db3325ac3e244919f866a2e04b9b9a0, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.841456 26486 consensus_queue.cc:260] T 3f58a7559c834237966455a46e2c6d02 P 9db3325ac3e244919f866a2e04b9b9a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.841538 26486 ts_tablet_manager.cc:1428] T 3f58a7559c834237966455a46e2c6d02 P 9db3325ac3e244919f866a2e04b9b9a0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.841604 26486 tablet_bootstrap.cc:492] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 9db3325ac3e244919f866a2e04b9b9a0: Bootstrap starting.
I20260808 07:03:29.841952 26486 tablet_bootstrap.cc:654] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 9db3325ac3e244919f866a2e04b9b9a0: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.838238 26127 tablet_service.cc:1467] Processing CreateTablet for tablet 55dd567790424028b86b3883a524be4a (DEFAULT_TABLE table=test-workload [id=bd244df9d0d14a4fa0c29579cec9d0a3]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260808 07:03:29.842116 26127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55dd567790424028b86b3883a524be4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.842489 26486 tablet_bootstrap.cc:492] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 9db3325ac3e244919f866a2e04b9b9a0: No bootstrap required, opened a new log
I20260808 07:03:29.842537 26486 ts_tablet_manager.cc:1397] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 9db3325ac3e244919f866a2e04b9b9a0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 07:03:29.842691 26485 tablet_bootstrap.cc:492] T 3f58a7559c834237966455a46e2c6d02 P b4106f2b661f4faaa255d5636a29788a: No bootstrap required, opened a new log
I20260808 07:03:29.842675 26486 raft_consensus.cc:348] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 9db3325ac3e244919f866a2e04b9b9a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.842733 26485 ts_tablet_manager.cc:1397] T 3f58a7559c834237966455a46e2c6d02 P b4106f2b661f4faaa255d5636a29788a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:03:29.842744 26486 raft_consensus.cc:729] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 9db3325ac3e244919f866a2e04b9b9a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db3325ac3e244919f866a2e04b9b9a0, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.842777 26486 consensus_queue.cc:260] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 9db3325ac3e244919f866a2e04b9b9a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.842869 26486 ts_tablet_manager.cc:1428] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 9db3325ac3e244919f866a2e04b9b9a0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.842880 26485 raft_consensus.cc:348] T 3f58a7559c834237966455a46e2c6d02 P b4106f2b661f4faaa255d5636a29788a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.842922 26486 tablet_bootstrap.cc:492] T 97901242097c446ebf1d046fe86d0800 P 9db3325ac3e244919f866a2e04b9b9a0: Bootstrap starting.
I20260808 07:03:29.842942 26485 raft_consensus.cc:729] T 3f58a7559c834237966455a46e2c6d02 P b4106f2b661f4faaa255d5636a29788a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4106f2b661f4faaa255d5636a29788a, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.842963 26485 consensus_queue.cc:260] T 3f58a7559c834237966455a46e2c6d02 P b4106f2b661f4faaa255d5636a29788a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.843045 26485 ts_tablet_manager.cc:1428] T 3f58a7559c834237966455a46e2c6d02 P b4106f2b661f4faaa255d5636a29788a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.843091 26485 tablet_bootstrap.cc:492] T 0080f8a2994e4252802ea6c5a6f4863b P b4106f2b661f4faaa255d5636a29788a: Bootstrap starting.
I20260808 07:03:29.843322 26486 tablet_bootstrap.cc:654] T 97901242097c446ebf1d046fe86d0800 P 9db3325ac3e244919f866a2e04b9b9a0: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.843390 26485 tablet_bootstrap.cc:654] T 0080f8a2994e4252802ea6c5a6f4863b P b4106f2b661f4faaa255d5636a29788a: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.843657 26262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0d8f0133ef04fbcb07c3e93ce4a8382. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.843755 26260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 051057f218174b81bf33ba8a5834e9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.843880 26256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1733039f2bdd4c4fa9a47522cbdefe9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.845511 26264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3f1db05fe244bae87a077fba49f764d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.845602 26263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97901242097c446ebf1d046fe86d0800. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.845691 26258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f58a7559c834237966455a46e2c6d02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.846107 26261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55dd567790424028b86b3883a524be4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.838076 26130 tablet_service.cc:1467] Processing CreateTablet for tablet c3f1db05fe244bae87a077fba49f764d (DEFAULT_TABLE table=test-workload [id=bd244df9d0d14a4fa0c29579cec9d0a3]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260808 07:03:29.846961 26130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3f1db05fe244bae87a077fba49f764d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.847065 26259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f64d7ac8e8894a81bbbe7d9f364f9ea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:03:29.849895 26502 tablet_bootstrap.cc:492] T 051057f218174b81bf33ba8a5834e9da P 117e71d831de49c89d100e8d430e6f91: Bootstrap starting.
I20260808 07:03:29.850695 26502 tablet_bootstrap.cc:654] T 051057f218174b81bf33ba8a5834e9da P 117e71d831de49c89d100e8d430e6f91: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.851045 26502 log.cc:824] T 051057f218174b81bf33ba8a5834e9da P 117e71d831de49c89d100e8d430e6f91: Log is configured to *not* fsync() on all Append() calls
I20260808 07:03:29.851634 26486 tablet_bootstrap.cc:492] T 97901242097c446ebf1d046fe86d0800 P 9db3325ac3e244919f866a2e04b9b9a0: No bootstrap required, opened a new log
I20260808 07:03:29.851702 26486 ts_tablet_manager.cc:1397] T 97901242097c446ebf1d046fe86d0800 P 9db3325ac3e244919f866a2e04b9b9a0: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260808 07:03:29.851768 26485 tablet_bootstrap.cc:492] T 0080f8a2994e4252802ea6c5a6f4863b P b4106f2b661f4faaa255d5636a29788a: No bootstrap required, opened a new log
I20260808 07:03:29.851830 26485 ts_tablet_manager.cc:1397] T 0080f8a2994e4252802ea6c5a6f4863b P b4106f2b661f4faaa255d5636a29788a: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260808 07:03:29.851847 26502 tablet_bootstrap.cc:492] T 051057f218174b81bf33ba8a5834e9da P 117e71d831de49c89d100e8d430e6f91: No bootstrap required, opened a new log
I20260808 07:03:29.851907 26502 ts_tablet_manager.cc:1397] T 051057f218174b81bf33ba8a5834e9da P 117e71d831de49c89d100e8d430e6f91: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 07:03:29.851866 26486 raft_consensus.cc:348] T 97901242097c446ebf1d046fe86d0800 P 9db3325ac3e244919f866a2e04b9b9a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.851948 26486 raft_consensus.cc:729] T 97901242097c446ebf1d046fe86d0800 P 9db3325ac3e244919f866a2e04b9b9a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db3325ac3e244919f866a2e04b9b9a0, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.851990 26486 consensus_queue.cc:260] T 97901242097c446ebf1d046fe86d0800 P 9db3325ac3e244919f866a2e04b9b9a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.852006 26485 raft_consensus.cc:348] T 0080f8a2994e4252802ea6c5a6f4863b P b4106f2b661f4faaa255d5636a29788a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.852083 26485 raft_consensus.cc:729] T 0080f8a2994e4252802ea6c5a6f4863b P b4106f2b661f4faaa255d5636a29788a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4106f2b661f4faaa255d5636a29788a, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.852082 26486 ts_tablet_manager.cc:1428] T 97901242097c446ebf1d046fe86d0800 P 9db3325ac3e244919f866a2e04b9b9a0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.852125 26485 consensus_queue.cc:260] T 0080f8a2994e4252802ea6c5a6f4863b P b4106f2b661f4faaa255d5636a29788a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.852200 26486 tablet_bootstrap.cc:492] T 051057f218174b81bf33ba8a5834e9da P 9db3325ac3e244919f866a2e04b9b9a0: Bootstrap starting.
I20260808 07:03:29.852222 26485 ts_tablet_manager.cc:1428] T 0080f8a2994e4252802ea6c5a6f4863b P b4106f2b661f4faaa255d5636a29788a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 07:03:29.852322 26485 tablet_bootstrap.cc:492] T 1733039f2bdd4c4fa9a47522cbdefe9e P b4106f2b661f4faaa255d5636a29788a: Bootstrap starting.
I20260808 07:03:29.852910 26486 tablet_bootstrap.cc:654] T 051057f218174b81bf33ba8a5834e9da P 9db3325ac3e244919f866a2e04b9b9a0: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.852911 26485 tablet_bootstrap.cc:654] T 1733039f2bdd4c4fa9a47522cbdefe9e P b4106f2b661f4faaa255d5636a29788a: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.853672 26485 tablet_bootstrap.cc:492] T 1733039f2bdd4c4fa9a47522cbdefe9e P b4106f2b661f4faaa255d5636a29788a: No bootstrap required, opened a new log
I20260808 07:03:29.853726 26485 ts_tablet_manager.cc:1397] T 1733039f2bdd4c4fa9a47522cbdefe9e P b4106f2b661f4faaa255d5636a29788a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 07:03:29.853853 26486 tablet_bootstrap.cc:492] T 051057f218174b81bf33ba8a5834e9da P 9db3325ac3e244919f866a2e04b9b9a0: No bootstrap required, opened a new log
I20260808 07:03:29.853868 26485 raft_consensus.cc:348] T 1733039f2bdd4c4fa9a47522cbdefe9e P b4106f2b661f4faaa255d5636a29788a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.853912 26486 ts_tablet_manager.cc:1397] T 051057f218174b81bf33ba8a5834e9da P 9db3325ac3e244919f866a2e04b9b9a0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:03:29.853952 26485 raft_consensus.cc:729] T 1733039f2bdd4c4fa9a47522cbdefe9e P b4106f2b661f4faaa255d5636a29788a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4106f2b661f4faaa255d5636a29788a, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.853987 26485 consensus_queue.cc:260] T 1733039f2bdd4c4fa9a47522cbdefe9e P b4106f2b661f4faaa255d5636a29788a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.854070 26485 ts_tablet_manager.cc:1428] T 1733039f2bdd4c4fa9a47522cbdefe9e P b4106f2b661f4faaa255d5636a29788a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 07:03:29.854074 26486 raft_consensus.cc:348] T 051057f218174b81bf33ba8a5834e9da P 9db3325ac3e244919f866a2e04b9b9a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.854142 26485 tablet_bootstrap.cc:492] T dad0a14869924e4a895536588073b33b P b4106f2b661f4faaa255d5636a29788a: Bootstrap starting.
I20260808 07:03:29.854152 26486 raft_consensus.cc:729] T 051057f218174b81bf33ba8a5834e9da P 9db3325ac3e244919f866a2e04b9b9a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db3325ac3e244919f866a2e04b9b9a0, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.854182 26486 consensus_queue.cc:260] T 051057f218174b81bf33ba8a5834e9da P 9db3325ac3e244919f866a2e04b9b9a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.854288 26486 ts_tablet_manager.cc:1428] T 051057f218174b81bf33ba8a5834e9da P 9db3325ac3e244919f866a2e04b9b9a0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.854375 26486 tablet_bootstrap.cc:492] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 9db3325ac3e244919f866a2e04b9b9a0: Bootstrap starting.
I20260808 07:03:29.854552 26485 tablet_bootstrap.cc:654] T dad0a14869924e4a895536588073b33b P b4106f2b661f4faaa255d5636a29788a: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.855361 26486 tablet_bootstrap.cc:654] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 9db3325ac3e244919f866a2e04b9b9a0: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.855661 26485 tablet_bootstrap.cc:492] T dad0a14869924e4a895536588073b33b P b4106f2b661f4faaa255d5636a29788a: No bootstrap required, opened a new log
I20260808 07:03:29.855711 26485 ts_tablet_manager.cc:1397] T dad0a14869924e4a895536588073b33b P b4106f2b661f4faaa255d5636a29788a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:03:29.855592 26502 raft_consensus.cc:348] T 051057f218174b81bf33ba8a5834e9da P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.855827 26502 raft_consensus.cc:729] T 051057f218174b81bf33ba8a5834e9da P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 117e71d831de49c89d100e8d430e6f91, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.855844 26485 raft_consensus.cc:348] T dad0a14869924e4a895536588073b33b P b4106f2b661f4faaa255d5636a29788a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } } peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } }
I20260808 07:03:29.855911 26485 raft_consensus.cc:729] T dad0a14869924e4a895536588073b33b P b4106f2b661f4faaa255d5636a29788a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4106f2b661f4faaa255d5636a29788a, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.855943 26485 consensus_queue.cc:260] T dad0a14869924e4a895536588073b33b P b4106f2b661f4faaa255d5636a29788a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } } peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } }
I20260808 07:03:29.855932 26502 consensus_queue.cc:260] T 051057f218174b81bf33ba8a5834e9da P 117e71d831de49c89d100e8d430e6f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.856073 26485 ts_tablet_manager.cc:1428] T dad0a14869924e4a895536588073b33b P b4106f2b661f4faaa255d5636a29788a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 07:03:29.856137 26485 tablet_bootstrap.cc:492] T c3f1db05fe244bae87a077fba49f764d P b4106f2b661f4faaa255d5636a29788a: Bootstrap starting.
I20260808 07:03:29.856220 26502 ts_tablet_manager.cc:1428] T 051057f218174b81bf33ba8a5834e9da P 117e71d831de49c89d100e8d430e6f91: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260808 07:03:29.856235 26332 heartbeater.cc:503] Master 127.15.250.190:38139 was elected leader, sending a full tablet report...
I20260808 07:03:29.856365 26502 tablet_bootstrap.cc:492] T 1733039f2bdd4c4fa9a47522cbdefe9e P 117e71d831de49c89d100e8d430e6f91: Bootstrap starting.
I20260808 07:03:29.856629 26486 tablet_bootstrap.cc:492] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 9db3325ac3e244919f866a2e04b9b9a0: No bootstrap required, opened a new log
I20260808 07:03:29.856729 26485 tablet_bootstrap.cc:654] T c3f1db05fe244bae87a077fba49f764d P b4106f2b661f4faaa255d5636a29788a: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.856757 26486 ts_tablet_manager.cc:1397] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 9db3325ac3e244919f866a2e04b9b9a0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:03:29.856820 26502 tablet_bootstrap.cc:654] T 1733039f2bdd4c4fa9a47522cbdefe9e P 117e71d831de49c89d100e8d430e6f91: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.857016 26486 raft_consensus.cc:348] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 9db3325ac3e244919f866a2e04b9b9a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.857169 26486 raft_consensus.cc:729] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 9db3325ac3e244919f866a2e04b9b9a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db3325ac3e244919f866a2e04b9b9a0, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.857270 26486 consensus_queue.cc:260] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 9db3325ac3e244919f866a2e04b9b9a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.857484 26486 ts_tablet_manager.cc:1428] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 9db3325ac3e244919f866a2e04b9b9a0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 07:03:29.857491 26502 tablet_bootstrap.cc:492] T 1733039f2bdd4c4fa9a47522cbdefe9e P 117e71d831de49c89d100e8d430e6f91: No bootstrap required, opened a new log
I20260808 07:03:29.857537 26502 ts_tablet_manager.cc:1397] T 1733039f2bdd4c4fa9a47522cbdefe9e P 117e71d831de49c89d100e8d430e6f91: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 07:03:29.857651 26486 tablet_bootstrap.cc:492] T 55dd567790424028b86b3883a524be4a P 9db3325ac3e244919f866a2e04b9b9a0: Bootstrap starting.
I20260808 07:03:29.857722 26502 raft_consensus.cc:348] T 1733039f2bdd4c4fa9a47522cbdefe9e P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.857807 26502 raft_consensus.cc:729] T 1733039f2bdd4c4fa9a47522cbdefe9e P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 117e71d831de49c89d100e8d430e6f91, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.857849 26502 consensus_queue.cc:260] T 1733039f2bdd4c4fa9a47522cbdefe9e P 117e71d831de49c89d100e8d430e6f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.857937 26502 ts_tablet_manager.cc:1428] T 1733039f2bdd4c4fa9a47522cbdefe9e P 117e71d831de49c89d100e8d430e6f91: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.858003 26502 tablet_bootstrap.cc:492] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 117e71d831de49c89d100e8d430e6f91: Bootstrap starting.
I20260808 07:03:29.858168 26486 tablet_bootstrap.cc:654] T 55dd567790424028b86b3883a524be4a P 9db3325ac3e244919f866a2e04b9b9a0: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.858453 26502 tablet_bootstrap.cc:654] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 117e71d831de49c89d100e8d430e6f91: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.858799 26486 tablet_bootstrap.cc:492] T 55dd567790424028b86b3883a524be4a P 9db3325ac3e244919f866a2e04b9b9a0: No bootstrap required, opened a new log
I20260808 07:03:29.858851 26486 ts_tablet_manager.cc:1397] T 55dd567790424028b86b3883a524be4a P 9db3325ac3e244919f866a2e04b9b9a0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 07:03:29.858989 26486 raft_consensus.cc:348] T 55dd567790424028b86b3883a524be4a P 9db3325ac3e244919f866a2e04b9b9a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.859066 26486 raft_consensus.cc:729] T 55dd567790424028b86b3883a524be4a P 9db3325ac3e244919f866a2e04b9b9a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db3325ac3e244919f866a2e04b9b9a0, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.859109 26502 tablet_bootstrap.cc:492] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 117e71d831de49c89d100e8d430e6f91: No bootstrap required, opened a new log
I20260808 07:03:29.859097 26486 consensus_queue.cc:260] T 55dd567790424028b86b3883a524be4a P 9db3325ac3e244919f866a2e04b9b9a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.859146 26502 ts_tablet_manager.cc:1397] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 117e71d831de49c89d100e8d430e6f91: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 07:03:29.859162 26485 tablet_bootstrap.cc:492] T c3f1db05fe244bae87a077fba49f764d P b4106f2b661f4faaa255d5636a29788a: No bootstrap required, opened a new log
I20260808 07:03:29.859200 26485 ts_tablet_manager.cc:1397] T c3f1db05fe244bae87a077fba49f764d P b4106f2b661f4faaa255d5636a29788a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260808 07:03:29.859215 26486 ts_tablet_manager.cc:1428] T 55dd567790424028b86b3883a524be4a P 9db3325ac3e244919f866a2e04b9b9a0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.859294 26486 tablet_bootstrap.cc:492] T c3f1db05fe244bae87a077fba49f764d P 9db3325ac3e244919f866a2e04b9b9a0: Bootstrap starting.
I20260808 07:03:29.859295 26502 raft_consensus.cc:348] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.859354 26502 raft_consensus.cc:729] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 117e71d831de49c89d100e8d430e6f91, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.859346 26485 raft_consensus.cc:348] T c3f1db05fe244bae87a077fba49f764d P b4106f2b661f4faaa255d5636a29788a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.859381 26502 consensus_queue.cc:260] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 117e71d831de49c89d100e8d430e6f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.859419 26485 raft_consensus.cc:729] T c3f1db05fe244bae87a077fba49f764d P b4106f2b661f4faaa255d5636a29788a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4106f2b661f4faaa255d5636a29788a, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.859449 26485 consensus_queue.cc:260] T c3f1db05fe244bae87a077fba49f764d P b4106f2b661f4faaa255d5636a29788a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.859472 26502 ts_tablet_manager.cc:1428] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 117e71d831de49c89d100e8d430e6f91: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.859520 26502 tablet_bootstrap.cc:492] T 0080f8a2994e4252802ea6c5a6f4863b P 117e71d831de49c89d100e8d430e6f91: Bootstrap starting.
I20260808 07:03:29.859553 26485 ts_tablet_manager.cc:1428] T c3f1db05fe244bae87a077fba49f764d P b4106f2b661f4faaa255d5636a29788a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.859606 26485 tablet_bootstrap.cc:492] T e0d8f0133ef04fbcb07c3e93ce4a8382 P b4106f2b661f4faaa255d5636a29788a: Bootstrap starting.
I20260808 07:03:29.859875 26486 tablet_bootstrap.cc:654] T c3f1db05fe244bae87a077fba49f764d P 9db3325ac3e244919f866a2e04b9b9a0: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.860041 26485 tablet_bootstrap.cc:654] T e0d8f0133ef04fbcb07c3e93ce4a8382 P b4106f2b661f4faaa255d5636a29788a: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.860105 26502 tablet_bootstrap.cc:654] T 0080f8a2994e4252802ea6c5a6f4863b P 117e71d831de49c89d100e8d430e6f91: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.860581 26486 tablet_bootstrap.cc:492] T c3f1db05fe244bae87a077fba49f764d P 9db3325ac3e244919f866a2e04b9b9a0: No bootstrap required, opened a new log
I20260808 07:03:29.860628 26486 ts_tablet_manager.cc:1397] T c3f1db05fe244bae87a077fba49f764d P 9db3325ac3e244919f866a2e04b9b9a0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 07:03:29.860669 26485 tablet_bootstrap.cc:492] T e0d8f0133ef04fbcb07c3e93ce4a8382 P b4106f2b661f4faaa255d5636a29788a: No bootstrap required, opened a new log
I20260808 07:03:29.860713 26485 ts_tablet_manager.cc:1397] T e0d8f0133ef04fbcb07c3e93ce4a8382 P b4106f2b661f4faaa255d5636a29788a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 07:03:29.860769 26486 raft_consensus.cc:348] T c3f1db05fe244bae87a077fba49f764d P 9db3325ac3e244919f866a2e04b9b9a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.860841 26486 raft_consensus.cc:729] T c3f1db05fe244bae87a077fba49f764d P 9db3325ac3e244919f866a2e04b9b9a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db3325ac3e244919f866a2e04b9b9a0, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.860869 26486 consensus_queue.cc:260] T c3f1db05fe244bae87a077fba49f764d P 9db3325ac3e244919f866a2e04b9b9a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.860862 26485 raft_consensus.cc:348] T e0d8f0133ef04fbcb07c3e93ce4a8382 P b4106f2b661f4faaa255d5636a29788a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.860927 26485 raft_consensus.cc:729] T e0d8f0133ef04fbcb07c3e93ce4a8382 P b4106f2b661f4faaa255d5636a29788a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4106f2b661f4faaa255d5636a29788a, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.860956 26486 ts_tablet_manager.cc:1428] T c3f1db05fe244bae87a077fba49f764d P 9db3325ac3e244919f866a2e04b9b9a0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.860960 26485 consensus_queue.cc:260] T e0d8f0133ef04fbcb07c3e93ce4a8382 P b4106f2b661f4faaa255d5636a29788a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.861069 26485 ts_tablet_manager.cc:1428] T e0d8f0133ef04fbcb07c3e93ce4a8382 P b4106f2b661f4faaa255d5636a29788a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.861897 26502 tablet_bootstrap.cc:492] T 0080f8a2994e4252802ea6c5a6f4863b P 117e71d831de49c89d100e8d430e6f91: No bootstrap required, opened a new log
I20260808 07:03:29.861946 26502 ts_tablet_manager.cc:1397] T 0080f8a2994e4252802ea6c5a6f4863b P 117e71d831de49c89d100e8d430e6f91: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 07:03:29.862107 26502 raft_consensus.cc:348] T 0080f8a2994e4252802ea6c5a6f4863b P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.862190 26502 raft_consensus.cc:729] T 0080f8a2994e4252802ea6c5a6f4863b P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 117e71d831de49c89d100e8d430e6f91, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.862231 26502 consensus_queue.cc:260] T 0080f8a2994e4252802ea6c5a6f4863b P 117e71d831de49c89d100e8d430e6f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.862318 26502 ts_tablet_manager.cc:1428] T 0080f8a2994e4252802ea6c5a6f4863b P 117e71d831de49c89d100e8d430e6f91: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.862382 26502 tablet_bootstrap.cc:492] T dad0a14869924e4a895536588073b33b P 117e71d831de49c89d100e8d430e6f91: Bootstrap starting.
I20260808 07:03:29.862818 26502 tablet_bootstrap.cc:654] T dad0a14869924e4a895536588073b33b P 117e71d831de49c89d100e8d430e6f91: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.863512 26502 tablet_bootstrap.cc:492] T dad0a14869924e4a895536588073b33b P 117e71d831de49c89d100e8d430e6f91: No bootstrap required, opened a new log
I20260808 07:03:29.863569 26502 ts_tablet_manager.cc:1397] T dad0a14869924e4a895536588073b33b P 117e71d831de49c89d100e8d430e6f91: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260808 07:03:29.863703 26502 raft_consensus.cc:348] T dad0a14869924e4a895536588073b33b P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } } peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } }
I20260808 07:03:29.863757 26502 raft_consensus.cc:729] T dad0a14869924e4a895536588073b33b P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 117e71d831de49c89d100e8d430e6f91, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.863811 26502 consensus_queue.cc:260] T dad0a14869924e4a895536588073b33b P 117e71d831de49c89d100e8d430e6f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } } peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } }
I20260808 07:03:29.863919 26502 ts_tablet_manager.cc:1428] T dad0a14869924e4a895536588073b33b P 117e71d831de49c89d100e8d430e6f91: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.863981 26502 tablet_bootstrap.cc:492] T 3f58a7559c834237966455a46e2c6d02 P 117e71d831de49c89d100e8d430e6f91: Bootstrap starting.
I20260808 07:03:29.864415 26502 tablet_bootstrap.cc:654] T 3f58a7559c834237966455a46e2c6d02 P 117e71d831de49c89d100e8d430e6f91: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.865015 26502 tablet_bootstrap.cc:492] T 3f58a7559c834237966455a46e2c6d02 P 117e71d831de49c89d100e8d430e6f91: No bootstrap required, opened a new log
I20260808 07:03:29.865054 26502 ts_tablet_manager.cc:1397] T 3f58a7559c834237966455a46e2c6d02 P 117e71d831de49c89d100e8d430e6f91: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260808 07:03:29.865211 26502 raft_consensus.cc:348] T 3f58a7559c834237966455a46e2c6d02 P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.865263 26502 raft_consensus.cc:729] T 3f58a7559c834237966455a46e2c6d02 P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 117e71d831de49c89d100e8d430e6f91, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.865284 26502 consensus_queue.cc:260] T 3f58a7559c834237966455a46e2c6d02 P 117e71d831de49c89d100e8d430e6f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.865404 26502 ts_tablet_manager.cc:1428] T 3f58a7559c834237966455a46e2c6d02 P 117e71d831de49c89d100e8d430e6f91: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.865469 26502 tablet_bootstrap.cc:492] T 97901242097c446ebf1d046fe86d0800 P 117e71d831de49c89d100e8d430e6f91: Bootstrap starting.
I20260808 07:03:29.865971 26502 tablet_bootstrap.cc:654] T 97901242097c446ebf1d046fe86d0800 P 117e71d831de49c89d100e8d430e6f91: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.866614 26502 tablet_bootstrap.cc:492] T 97901242097c446ebf1d046fe86d0800 P 117e71d831de49c89d100e8d430e6f91: No bootstrap required, opened a new log
I20260808 07:03:29.866668 26502 ts_tablet_manager.cc:1397] T 97901242097c446ebf1d046fe86d0800 P 117e71d831de49c89d100e8d430e6f91: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260808 07:03:29.866856 26502 raft_consensus.cc:348] T 97901242097c446ebf1d046fe86d0800 P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.866933 26502 raft_consensus.cc:729] T 97901242097c446ebf1d046fe86d0800 P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 117e71d831de49c89d100e8d430e6f91, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.866962 26502 consensus_queue.cc:260] T 97901242097c446ebf1d046fe86d0800 P 117e71d831de49c89d100e8d430e6f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.867050 26502 ts_tablet_manager.cc:1428] T 97901242097c446ebf1d046fe86d0800 P 117e71d831de49c89d100e8d430e6f91: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.867451 26502 tablet_bootstrap.cc:492] T 55dd567790424028b86b3883a524be4a P 117e71d831de49c89d100e8d430e6f91: Bootstrap starting.
I20260808 07:03:29.867895 26502 tablet_bootstrap.cc:654] T 55dd567790424028b86b3883a524be4a P 117e71d831de49c89d100e8d430e6f91: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.868551 26502 tablet_bootstrap.cc:492] T 55dd567790424028b86b3883a524be4a P 117e71d831de49c89d100e8d430e6f91: No bootstrap required, opened a new log
I20260808 07:03:29.868592 26502 ts_tablet_manager.cc:1397] T 55dd567790424028b86b3883a524be4a P 117e71d831de49c89d100e8d430e6f91: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260808 07:03:29.868763 26502 raft_consensus.cc:348] T 55dd567790424028b86b3883a524be4a P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.868817 26502 raft_consensus.cc:729] T 55dd567790424028b86b3883a524be4a P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 117e71d831de49c89d100e8d430e6f91, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.868836 26502 consensus_queue.cc:260] T 55dd567790424028b86b3883a524be4a P 117e71d831de49c89d100e8d430e6f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.868955 26502 ts_tablet_manager.cc:1428] T 55dd567790424028b86b3883a524be4a P 117e71d831de49c89d100e8d430e6f91: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.869024 26502 tablet_bootstrap.cc:492] T c3f1db05fe244bae87a077fba49f764d P 117e71d831de49c89d100e8d430e6f91: Bootstrap starting.
I20260808 07:03:29.869451 26502 tablet_bootstrap.cc:654] T c3f1db05fe244bae87a077fba49f764d P 117e71d831de49c89d100e8d430e6f91: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.870076 26502 tablet_bootstrap.cc:492] T c3f1db05fe244bae87a077fba49f764d P 117e71d831de49c89d100e8d430e6f91: No bootstrap required, opened a new log
I20260808 07:03:29.870118 26502 ts_tablet_manager.cc:1397] T c3f1db05fe244bae87a077fba49f764d P 117e71d831de49c89d100e8d430e6f91: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260808 07:03:29.870220 26502 raft_consensus.cc:348] T c3f1db05fe244bae87a077fba49f764d P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.870275 26502 raft_consensus.cc:729] T c3f1db05fe244bae87a077fba49f764d P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 117e71d831de49c89d100e8d430e6f91, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.870292 26502 consensus_queue.cc:260] T c3f1db05fe244bae87a077fba49f764d P 117e71d831de49c89d100e8d430e6f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.870360 26502 ts_tablet_manager.cc:1428] T c3f1db05fe244bae87a077fba49f764d P 117e71d831de49c89d100e8d430e6f91: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.870438 26502 tablet_bootstrap.cc:492] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 117e71d831de49c89d100e8d430e6f91: Bootstrap starting.
I20260808 07:03:29.870924 26502 tablet_bootstrap.cc:654] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 117e71d831de49c89d100e8d430e6f91: Neither blocks nor log segments found. Creating new log.
I20260808 07:03:29.871472 26502 tablet_bootstrap.cc:492] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 117e71d831de49c89d100e8d430e6f91: No bootstrap required, opened a new log
I20260808 07:03:29.871510 26502 ts_tablet_manager.cc:1397] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 117e71d831de49c89d100e8d430e6f91: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260808 07:03:29.871660 26502 raft_consensus.cc:348] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.871711 26502 raft_consensus.cc:729] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 117e71d831de49c89d100e8d430e6f91, State: Initialized, Role: FOLLOWER
I20260808 07:03:29.871757 26502 consensus_queue.cc:260] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 117e71d831de49c89d100e8d430e6f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.871850 26502 ts_tablet_manager.cc:1428] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 117e71d831de49c89d100e8d430e6f91: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 07:03:29.886482 26286 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c3f1db05fe244bae87a077fba49f764d"
dest_uuid: "117e71d831de49c89d100e8d430e6f91"
from {username='slave'} at 127.0.0.1:43438
I20260808 07:03:29.886613 26286 raft_consensus.cc:482] T c3f1db05fe244bae87a077fba49f764d P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 07:03:29.886656 26286 raft_consensus.cc:3037] T c3f1db05fe244bae87a077fba49f764d P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.887346 26286 raft_consensus.cc:504] T c3f1db05fe244bae87a077fba49f764d P 117e71d831de49c89d100e8d430e6f91 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.887722 26286 leader_election.cc:288] T c3f1db05fe244bae87a077fba49f764d P 117e71d831de49c89d100e8d430e6f91 [CANDIDATE]: Term 1 election: Requested vote from peers 9db3325ac3e244919f866a2e04b9b9a0 (127.15.250.129:42123), b4106f2b661f4faaa255d5636a29788a (127.15.250.131:34037)
I20260808 07:03:29.891273 26151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f1db05fe244bae87a077fba49f764d" candidate_uuid: "117e71d831de49c89d100e8d430e6f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9db3325ac3e244919f866a2e04b9b9a0"
I20260808 07:03:29.891373 26419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f1db05fe244bae87a077fba49f764d" candidate_uuid: "117e71d831de49c89d100e8d430e6f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b4106f2b661f4faaa255d5636a29788a"
I20260808 07:03:29.891445 26151 raft_consensus.cc:3037] T c3f1db05fe244bae87a077fba49f764d P 9db3325ac3e244919f866a2e04b9b9a0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.891518 26419 raft_consensus.cc:3037] T c3f1db05fe244bae87a077fba49f764d P b4106f2b661f4faaa255d5636a29788a [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.892355 26151 raft_consensus.cc:2454] T c3f1db05fe244bae87a077fba49f764d P 9db3325ac3e244919f866a2e04b9b9a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 117e71d831de49c89d100e8d430e6f91 in term 1.
I20260808 07:03:29.892426 26419 raft_consensus.cc:2454] T c3f1db05fe244bae87a077fba49f764d P b4106f2b661f4faaa255d5636a29788a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 117e71d831de49c89d100e8d430e6f91 in term 1.
I20260808 07:03:29.892653 26220 leader_election.cc:302] T c3f1db05fe244bae87a077fba49f764d P 117e71d831de49c89d100e8d430e6f91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 117e71d831de49c89d100e8d430e6f91, 9db3325ac3e244919f866a2e04b9b9a0; no voters:
I20260808 07:03:29.892853 26508 raft_consensus.cc:2781] T c3f1db05fe244bae87a077fba49f764d P 117e71d831de49c89d100e8d430e6f91 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:29.893018 26508 raft_consensus.cc:686] T c3f1db05fe244bae87a077fba49f764d P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: Becoming Leader. State: Replica: 117e71d831de49c89d100e8d430e6f91, State: Running, Role: LEADER
I20260808 07:03:29.893162 26508 consensus_queue.cc:237] T c3f1db05fe244bae87a077fba49f764d P 117e71d831de49c89d100e8d430e6f91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.894201 26006 catalog_manager.cc:5495] T c3f1db05fe244bae87a077fba49f764d P 117e71d831de49c89d100e8d430e6f91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 117e71d831de49c89d100e8d430e6f91 (127.15.250.130). New cstate: current_term: 1 leader_uuid: "117e71d831de49c89d100e8d430e6f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } health_report { overall_health: UNKNOWN } } }
I20260808 07:03:29.898559 26286 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "55dd567790424028b86b3883a524be4a"
dest_uuid: "117e71d831de49c89d100e8d430e6f91"
from {username='slave'} at 127.0.0.1:43438
I20260808 07:03:29.898650 26286 raft_consensus.cc:482] T 55dd567790424028b86b3883a524be4a P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 07:03:29.898674 26286 raft_consensus.cc:3037] T 55dd567790424028b86b3883a524be4a P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.899317 26286 raft_consensus.cc:504] T 55dd567790424028b86b3883a524be4a P 117e71d831de49c89d100e8d430e6f91 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.899464 26286 leader_election.cc:288] T 55dd567790424028b86b3883a524be4a P 117e71d831de49c89d100e8d430e6f91 [CANDIDATE]: Term 1 election: Requested vote from peers 9db3325ac3e244919f866a2e04b9b9a0 (127.15.250.129:42123), b4106f2b661f4faaa255d5636a29788a (127.15.250.131:34037)
I20260808 07:03:29.899742 26151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55dd567790424028b86b3883a524be4a" candidate_uuid: "117e71d831de49c89d100e8d430e6f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9db3325ac3e244919f866a2e04b9b9a0"
I20260808 07:03:29.899765 26419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55dd567790424028b86b3883a524be4a" candidate_uuid: "117e71d831de49c89d100e8d430e6f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b4106f2b661f4faaa255d5636a29788a"
I20260808 07:03:29.899873 26419 raft_consensus.cc:3037] T 55dd567790424028b86b3883a524be4a P b4106f2b661f4faaa255d5636a29788a [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.899875 26151 raft_consensus.cc:3037] T 55dd567790424028b86b3883a524be4a P 9db3325ac3e244919f866a2e04b9b9a0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.900588 26419 raft_consensus.cc:2454] T 55dd567790424028b86b3883a524be4a P b4106f2b661f4faaa255d5636a29788a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 117e71d831de49c89d100e8d430e6f91 in term 1.
I20260808 07:03:29.900643 26151 raft_consensus.cc:2454] T 55dd567790424028b86b3883a524be4a P 9db3325ac3e244919f866a2e04b9b9a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 117e71d831de49c89d100e8d430e6f91 in term 1.
I20260808 07:03:29.900790 26220 leader_election.cc:302] T 55dd567790424028b86b3883a524be4a P 117e71d831de49c89d100e8d430e6f91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 117e71d831de49c89d100e8d430e6f91, b4106f2b661f4faaa255d5636a29788a; no voters:
I20260808 07:03:29.900892 26508 raft_consensus.cc:2781] T 55dd567790424028b86b3883a524be4a P 117e71d831de49c89d100e8d430e6f91 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:29.900947 26508 raft_consensus.cc:686] T 55dd567790424028b86b3883a524be4a P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: Becoming Leader. State: Replica: 117e71d831de49c89d100e8d430e6f91, State: Running, Role: LEADER
I20260808 07:03:29.900998 26508 consensus_queue.cc:237] T 55dd567790424028b86b3883a524be4a P 117e71d831de49c89d100e8d430e6f91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.901690 26006 catalog_manager.cc:5495] T 55dd567790424028b86b3883a524be4a P 117e71d831de49c89d100e8d430e6f91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 117e71d831de49c89d100e8d430e6f91 (127.15.250.130). New cstate: current_term: 1 leader_uuid: "117e71d831de49c89d100e8d430e6f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } health_report { overall_health: UNKNOWN } } }
I20260808 07:03:29.902982 26286 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e0d8f0133ef04fbcb07c3e93ce4a8382"
dest_uuid: "117e71d831de49c89d100e8d430e6f91"
from {username='slave'} at 127.0.0.1:43438
I20260808 07:03:29.903072 26286 raft_consensus.cc:482] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 07:03:29.903105 26286 raft_consensus.cc:3037] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.903731 26286 raft_consensus.cc:504] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 117e71d831de49c89d100e8d430e6f91 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.903877 26286 leader_election.cc:288] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 117e71d831de49c89d100e8d430e6f91 [CANDIDATE]: Term 1 election: Requested vote from peers 9db3325ac3e244919f866a2e04b9b9a0 (127.15.250.129:42123), b4106f2b661f4faaa255d5636a29788a (127.15.250.131:34037)
I20260808 07:03:29.904062 26151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0d8f0133ef04fbcb07c3e93ce4a8382" candidate_uuid: "117e71d831de49c89d100e8d430e6f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9db3325ac3e244919f866a2e04b9b9a0"
I20260808 07:03:29.904080 26419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0d8f0133ef04fbcb07c3e93ce4a8382" candidate_uuid: "117e71d831de49c89d100e8d430e6f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b4106f2b661f4faaa255d5636a29788a"
I20260808 07:03:29.904158 26419 raft_consensus.cc:3037] T e0d8f0133ef04fbcb07c3e93ce4a8382 P b4106f2b661f4faaa255d5636a29788a [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.904160 26151 raft_consensus.cc:3037] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 9db3325ac3e244919f866a2e04b9b9a0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.904932 26151 raft_consensus.cc:2454] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 9db3325ac3e244919f866a2e04b9b9a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 117e71d831de49c89d100e8d430e6f91 in term 1.
I20260808 07:03:29.904932 26419 raft_consensus.cc:2454] T e0d8f0133ef04fbcb07c3e93ce4a8382 P b4106f2b661f4faaa255d5636a29788a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 117e71d831de49c89d100e8d430e6f91 in term 1.
I20260808 07:03:29.905129 26220 leader_election.cc:302] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 117e71d831de49c89d100e8d430e6f91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 117e71d831de49c89d100e8d430e6f91, b4106f2b661f4faaa255d5636a29788a; no voters:
I20260808 07:03:29.905222 26508 raft_consensus.cc:2781] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 117e71d831de49c89d100e8d430e6f91 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:29.905277 26508 raft_consensus.cc:686] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: Becoming Leader. State: Replica: 117e71d831de49c89d100e8d430e6f91, State: Running, Role: LEADER
I20260808 07:03:29.905344 26508 consensus_queue.cc:237] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 117e71d831de49c89d100e8d430e6f91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.905932 26006 catalog_manager.cc:5495] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 117e71d831de49c89d100e8d430e6f91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 117e71d831de49c89d100e8d430e6f91 (127.15.250.130). New cstate: current_term: 1 leader_uuid: "117e71d831de49c89d100e8d430e6f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } health_report { overall_health: UNKNOWN } } }
I20260808 07:03:29.907140 26286 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "051057f218174b81bf33ba8a5834e9da"
dest_uuid: "117e71d831de49c89d100e8d430e6f91"
from {username='slave'} at 127.0.0.1:43438
I20260808 07:03:29.907213 26286 raft_consensus.cc:482] T 051057f218174b81bf33ba8a5834e9da P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 07:03:29.907234 26286 raft_consensus.cc:3037] T 051057f218174b81bf33ba8a5834e9da P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.907749 26286 raft_consensus.cc:504] T 051057f218174b81bf33ba8a5834e9da P 117e71d831de49c89d100e8d430e6f91 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.907879 26286 leader_election.cc:288] T 051057f218174b81bf33ba8a5834e9da P 117e71d831de49c89d100e8d430e6f91 [CANDIDATE]: Term 1 election: Requested vote from peers 9db3325ac3e244919f866a2e04b9b9a0 (127.15.250.129:42123), b4106f2b661f4faaa255d5636a29788a (127.15.250.131:34037)
I20260808 07:03:29.908084 26151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "051057f218174b81bf33ba8a5834e9da" candidate_uuid: "117e71d831de49c89d100e8d430e6f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9db3325ac3e244919f866a2e04b9b9a0"
I20260808 07:03:29.908170 26151 raft_consensus.cc:3037] T 051057f218174b81bf33ba8a5834e9da P 9db3325ac3e244919f866a2e04b9b9a0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.908138 26419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "051057f218174b81bf33ba8a5834e9da" candidate_uuid: "117e71d831de49c89d100e8d430e6f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b4106f2b661f4faaa255d5636a29788a"
I20260808 07:03:29.908285 26419 raft_consensus.cc:3037] T 051057f218174b81bf33ba8a5834e9da P b4106f2b661f4faaa255d5636a29788a [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.908802 26151 raft_consensus.cc:2454] T 051057f218174b81bf33ba8a5834e9da P 9db3325ac3e244919f866a2e04b9b9a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 117e71d831de49c89d100e8d430e6f91 in term 1.
I20260808 07:03:29.908924 26419 raft_consensus.cc:2454] T 051057f218174b81bf33ba8a5834e9da P b4106f2b661f4faaa255d5636a29788a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 117e71d831de49c89d100e8d430e6f91 in term 1.
I20260808 07:03:29.908974 26220 leader_election.cc:302] T 051057f218174b81bf33ba8a5834e9da P 117e71d831de49c89d100e8d430e6f91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 117e71d831de49c89d100e8d430e6f91, 9db3325ac3e244919f866a2e04b9b9a0; no voters:
I20260808 07:03:29.909055 26508 raft_consensus.cc:2781] T 051057f218174b81bf33ba8a5834e9da P 117e71d831de49c89d100e8d430e6f91 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:29.909111 26508 raft_consensus.cc:686] T 051057f218174b81bf33ba8a5834e9da P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: Becoming Leader. State: Replica: 117e71d831de49c89d100e8d430e6f91, State: Running, Role: LEADER
I20260808 07:03:29.909159 26508 consensus_queue.cc:237] T 051057f218174b81bf33ba8a5834e9da P 117e71d831de49c89d100e8d430e6f91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.909968 26006 catalog_manager.cc:5495] T 051057f218174b81bf33ba8a5834e9da P 117e71d831de49c89d100e8d430e6f91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 117e71d831de49c89d100e8d430e6f91 (127.15.250.130). New cstate: current_term: 1 leader_uuid: "117e71d831de49c89d100e8d430e6f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } health_report { overall_health: UNKNOWN } } }
W20260808 07:03:29.910367 26333 tablet.cc:2369] T 0080f8a2994e4252802ea6c5a6f4863b P 117e71d831de49c89d100e8d430e6f91: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 07:03:29.911327 26286 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dad0a14869924e4a895536588073b33b"
dest_uuid: "117e71d831de49c89d100e8d430e6f91"
from {username='slave'} at 127.0.0.1:43438
I20260808 07:03:29.911410 26286 raft_consensus.cc:482] T dad0a14869924e4a895536588073b33b P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 07:03:29.911442 26286 raft_consensus.cc:3037] T dad0a14869924e4a895536588073b33b P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.912050 26286 raft_consensus.cc:504] T dad0a14869924e4a895536588073b33b P 117e71d831de49c89d100e8d430e6f91 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } } peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } }
I20260808 07:03:29.912184 26286 leader_election.cc:288] T dad0a14869924e4a895536588073b33b P 117e71d831de49c89d100e8d430e6f91 [CANDIDATE]: Term 1 election: Requested vote from peers b4106f2b661f4faaa255d5636a29788a (127.15.250.131:34037), 9db3325ac3e244919f866a2e04b9b9a0 (127.15.250.129:42123)
I20260808 07:03:29.912418 26419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad0a14869924e4a895536588073b33b" candidate_uuid: "117e71d831de49c89d100e8d430e6f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b4106f2b661f4faaa255d5636a29788a"
I20260808 07:03:29.912504 26419 raft_consensus.cc:3037] T dad0a14869924e4a895536588073b33b P b4106f2b661f4faaa255d5636a29788a [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.912467 26151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad0a14869924e4a895536588073b33b" candidate_uuid: "117e71d831de49c89d100e8d430e6f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9db3325ac3e244919f866a2e04b9b9a0"
I20260808 07:03:29.912554 26151 raft_consensus.cc:3037] T dad0a14869924e4a895536588073b33b P 9db3325ac3e244919f866a2e04b9b9a0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.913203 26151 raft_consensus.cc:2454] T dad0a14869924e4a895536588073b33b P 9db3325ac3e244919f866a2e04b9b9a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 117e71d831de49c89d100e8d430e6f91 in term 1.
I20260808 07:03:29.913204 26419 raft_consensus.cc:2454] T dad0a14869924e4a895536588073b33b P b4106f2b661f4faaa255d5636a29788a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 117e71d831de49c89d100e8d430e6f91 in term 1.
I20260808 07:03:29.913388 26220 leader_election.cc:302] T dad0a14869924e4a895536588073b33b P 117e71d831de49c89d100e8d430e6f91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 117e71d831de49c89d100e8d430e6f91, b4106f2b661f4faaa255d5636a29788a; no voters:
I20260808 07:03:29.913482 26508 raft_consensus.cc:2781] T dad0a14869924e4a895536588073b33b P 117e71d831de49c89d100e8d430e6f91 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:29.913538 26508 raft_consensus.cc:686] T dad0a14869924e4a895536588073b33b P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: Becoming Leader. State: Replica: 117e71d831de49c89d100e8d430e6f91, State: Running, Role: LEADER
I20260808 07:03:29.913604 26508 consensus_queue.cc:237] T dad0a14869924e4a895536588073b33b P 117e71d831de49c89d100e8d430e6f91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } } peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } }
I20260808 07:03:29.914250 26006 catalog_manager.cc:5495] T dad0a14869924e4a895536588073b33b P 117e71d831de49c89d100e8d430e6f91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 117e71d831de49c89d100e8d430e6f91 (127.15.250.130). New cstate: current_term: 1 leader_uuid: "117e71d831de49c89d100e8d430e6f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } health_report { overall_health: HEALTHY } } }
I20260808 07:03:29.915555 26286 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1733039f2bdd4c4fa9a47522cbdefe9e"
dest_uuid: "117e71d831de49c89d100e8d430e6f91"
from {username='slave'} at 127.0.0.1:43438
I20260808 07:03:29.915623 26286 raft_consensus.cc:482] T 1733039f2bdd4c4fa9a47522cbdefe9e P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 07:03:29.915644 26286 raft_consensus.cc:3037] T 1733039f2bdd4c4fa9a47522cbdefe9e P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.916097 26286 raft_consensus.cc:504] T 1733039f2bdd4c4fa9a47522cbdefe9e P 117e71d831de49c89d100e8d430e6f91 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.916210 26286 leader_election.cc:288] T 1733039f2bdd4c4fa9a47522cbdefe9e P 117e71d831de49c89d100e8d430e6f91 [CANDIDATE]: Term 1 election: Requested vote from peers 9db3325ac3e244919f866a2e04b9b9a0 (127.15.250.129:42123), b4106f2b661f4faaa255d5636a29788a (127.15.250.131:34037)
I20260808 07:03:29.916383 26151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1733039f2bdd4c4fa9a47522cbdefe9e" candidate_uuid: "117e71d831de49c89d100e8d430e6f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9db3325ac3e244919f866a2e04b9b9a0"
I20260808 07:03:29.916476 26151 raft_consensus.cc:3037] T 1733039f2bdd4c4fa9a47522cbdefe9e P 9db3325ac3e244919f866a2e04b9b9a0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.916528 26419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1733039f2bdd4c4fa9a47522cbdefe9e" candidate_uuid: "117e71d831de49c89d100e8d430e6f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b4106f2b661f4faaa255d5636a29788a"
I20260808 07:03:29.916599 26419 raft_consensus.cc:3037] T 1733039f2bdd4c4fa9a47522cbdefe9e P b4106f2b661f4faaa255d5636a29788a [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.917131 26151 raft_consensus.cc:2454] T 1733039f2bdd4c4fa9a47522cbdefe9e P 9db3325ac3e244919f866a2e04b9b9a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 117e71d831de49c89d100e8d430e6f91 in term 1.
I20260808 07:03:29.917250 26419 raft_consensus.cc:2454] T 1733039f2bdd4c4fa9a47522cbdefe9e P b4106f2b661f4faaa255d5636a29788a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 117e71d831de49c89d100e8d430e6f91 in term 1.
I20260808 07:03:29.917294 26220 leader_election.cc:302] T 1733039f2bdd4c4fa9a47522cbdefe9e P 117e71d831de49c89d100e8d430e6f91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 117e71d831de49c89d100e8d430e6f91, 9db3325ac3e244919f866a2e04b9b9a0; no voters:
I20260808 07:03:29.917388 26508 raft_consensus.cc:2781] T 1733039f2bdd4c4fa9a47522cbdefe9e P 117e71d831de49c89d100e8d430e6f91 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:29.917469 26508 raft_consensus.cc:686] T 1733039f2bdd4c4fa9a47522cbdefe9e P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: Becoming Leader. State: Replica: 117e71d831de49c89d100e8d430e6f91, State: Running, Role: LEADER
I20260808 07:03:29.917527 26508 consensus_queue.cc:237] T 1733039f2bdd4c4fa9a47522cbdefe9e P 117e71d831de49c89d100e8d430e6f91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.918090 26006 catalog_manager.cc:5495] T 1733039f2bdd4c4fa9a47522cbdefe9e P 117e71d831de49c89d100e8d430e6f91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 117e71d831de49c89d100e8d430e6f91 (127.15.250.130). New cstate: current_term: 1 leader_uuid: "117e71d831de49c89d100e8d430e6f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } health_report { overall_health: UNKNOWN } } }
I20260808 07:03:29.919610 26286 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0080f8a2994e4252802ea6c5a6f4863b"
dest_uuid: "117e71d831de49c89d100e8d430e6f91"
from {username='slave'} at 127.0.0.1:43438
I20260808 07:03:29.919665 26286 raft_consensus.cc:482] T 0080f8a2994e4252802ea6c5a6f4863b P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 07:03:29.919683 26286 raft_consensus.cc:3037] T 0080f8a2994e4252802ea6c5a6f4863b P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.920171 26286 raft_consensus.cc:504] T 0080f8a2994e4252802ea6c5a6f4863b P 117e71d831de49c89d100e8d430e6f91 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.920331 26286 leader_election.cc:288] T 0080f8a2994e4252802ea6c5a6f4863b P 117e71d831de49c89d100e8d430e6f91 [CANDIDATE]: Term 1 election: Requested vote from peers 9db3325ac3e244919f866a2e04b9b9a0 (127.15.250.129:42123), b4106f2b661f4faaa255d5636a29788a (127.15.250.131:34037)
I20260808 07:03:29.920476 26419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0080f8a2994e4252802ea6c5a6f4863b" candidate_uuid: "117e71d831de49c89d100e8d430e6f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b4106f2b661f4faaa255d5636a29788a"
I20260808 07:03:29.920481 26151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0080f8a2994e4252802ea6c5a6f4863b" candidate_uuid: "117e71d831de49c89d100e8d430e6f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9db3325ac3e244919f866a2e04b9b9a0"
I20260808 07:03:29.920552 26419 raft_consensus.cc:3037] T 0080f8a2994e4252802ea6c5a6f4863b P b4106f2b661f4faaa255d5636a29788a [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.920554 26151 raft_consensus.cc:3037] T 0080f8a2994e4252802ea6c5a6f4863b P 9db3325ac3e244919f866a2e04b9b9a0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.921103 26151 raft_consensus.cc:2454] T 0080f8a2994e4252802ea6c5a6f4863b P 9db3325ac3e244919f866a2e04b9b9a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 117e71d831de49c89d100e8d430e6f91 in term 1.
I20260808 07:03:29.921104 26419 raft_consensus.cc:2454] T 0080f8a2994e4252802ea6c5a6f4863b P b4106f2b661f4faaa255d5636a29788a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 117e71d831de49c89d100e8d430e6f91 in term 1.
I20260808 07:03:29.921253 26220 leader_election.cc:302] T 0080f8a2994e4252802ea6c5a6f4863b P 117e71d831de49c89d100e8d430e6f91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 117e71d831de49c89d100e8d430e6f91, b4106f2b661f4faaa255d5636a29788a; no voters:
I20260808 07:03:29.921362 26508 raft_consensus.cc:2781] T 0080f8a2994e4252802ea6c5a6f4863b P 117e71d831de49c89d100e8d430e6f91 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:29.921425 26508 raft_consensus.cc:686] T 0080f8a2994e4252802ea6c5a6f4863b P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: Becoming Leader. State: Replica: 117e71d831de49c89d100e8d430e6f91, State: Running, Role: LEADER
I20260808 07:03:29.921471 26508 consensus_queue.cc:237] T 0080f8a2994e4252802ea6c5a6f4863b P 117e71d831de49c89d100e8d430e6f91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.922051 26006 catalog_manager.cc:5495] T 0080f8a2994e4252802ea6c5a6f4863b P 117e71d831de49c89d100e8d430e6f91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 117e71d831de49c89d100e8d430e6f91 (127.15.250.130). New cstate: current_term: 1 leader_uuid: "117e71d831de49c89d100e8d430e6f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } health_report { overall_health: UNKNOWN } } }
I20260808 07:03:29.923319 26286 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3f58a7559c834237966455a46e2c6d02"
dest_uuid: "117e71d831de49c89d100e8d430e6f91"
from {username='slave'} at 127.0.0.1:43438
I20260808 07:03:29.923385 26286 raft_consensus.cc:482] T 3f58a7559c834237966455a46e2c6d02 P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 07:03:29.923404 26286 raft_consensus.cc:3037] T 3f58a7559c834237966455a46e2c6d02 P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.923938 26286 raft_consensus.cc:504] T 3f58a7559c834237966455a46e2c6d02 P 117e71d831de49c89d100e8d430e6f91 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.924054 26286 leader_election.cc:288] T 3f58a7559c834237966455a46e2c6d02 P 117e71d831de49c89d100e8d430e6f91 [CANDIDATE]: Term 1 election: Requested vote from peers 9db3325ac3e244919f866a2e04b9b9a0 (127.15.250.129:42123), b4106f2b661f4faaa255d5636a29788a (127.15.250.131:34037)
I20260808 07:03:29.924244 26419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f58a7559c834237966455a46e2c6d02" candidate_uuid: "117e71d831de49c89d100e8d430e6f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b4106f2b661f4faaa255d5636a29788a"
I20260808 07:03:29.924225 26151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f58a7559c834237966455a46e2c6d02" candidate_uuid: "117e71d831de49c89d100e8d430e6f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9db3325ac3e244919f866a2e04b9b9a0"
I20260808 07:03:29.924347 26419 raft_consensus.cc:3037] T 3f58a7559c834237966455a46e2c6d02 P b4106f2b661f4faaa255d5636a29788a [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.924352 26151 raft_consensus.cc:3037] T 3f58a7559c834237966455a46e2c6d02 P 9db3325ac3e244919f866a2e04b9b9a0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.924856 26151 raft_consensus.cc:2454] T 3f58a7559c834237966455a46e2c6d02 P 9db3325ac3e244919f866a2e04b9b9a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 117e71d831de49c89d100e8d430e6f91 in term 1.
I20260808 07:03:29.924886 26419 raft_consensus.cc:2454] T 3f58a7559c834237966455a46e2c6d02 P b4106f2b661f4faaa255d5636a29788a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 117e71d831de49c89d100e8d430e6f91 in term 1.
I20260808 07:03:29.925025 26220 leader_election.cc:302] T 3f58a7559c834237966455a46e2c6d02 P 117e71d831de49c89d100e8d430e6f91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 117e71d831de49c89d100e8d430e6f91, b4106f2b661f4faaa255d5636a29788a; no voters:
I20260808 07:03:29.925128 26508 raft_consensus.cc:2781] T 3f58a7559c834237966455a46e2c6d02 P 117e71d831de49c89d100e8d430e6f91 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:29.925185 26508 raft_consensus.cc:686] T 3f58a7559c834237966455a46e2c6d02 P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: Becoming Leader. State: Replica: 117e71d831de49c89d100e8d430e6f91, State: Running, Role: LEADER
I20260808 07:03:29.925235 26508 consensus_queue.cc:237] T 3f58a7559c834237966455a46e2c6d02 P 117e71d831de49c89d100e8d430e6f91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.925771 26006 catalog_manager.cc:5495] T 3f58a7559c834237966455a46e2c6d02 P 117e71d831de49c89d100e8d430e6f91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 117e71d831de49c89d100e8d430e6f91 (127.15.250.130). New cstate: current_term: 1 leader_uuid: "117e71d831de49c89d100e8d430e6f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } health_report { overall_health: UNKNOWN } } }
I20260808 07:03:29.926999 26286 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "97901242097c446ebf1d046fe86d0800"
dest_uuid: "117e71d831de49c89d100e8d430e6f91"
from {username='slave'} at 127.0.0.1:43438
I20260808 07:03:29.927064 26286 raft_consensus.cc:482] T 97901242097c446ebf1d046fe86d0800 P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 07:03:29.927103 26286 raft_consensus.cc:3037] T 97901242097c446ebf1d046fe86d0800 P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.927558 26286 raft_consensus.cc:504] T 97901242097c446ebf1d046fe86d0800 P 117e71d831de49c89d100e8d430e6f91 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.927667 26286 leader_election.cc:288] T 97901242097c446ebf1d046fe86d0800 P 117e71d831de49c89d100e8d430e6f91 [CANDIDATE]: Term 1 election: Requested vote from peers 9db3325ac3e244919f866a2e04b9b9a0 (127.15.250.129:42123), b4106f2b661f4faaa255d5636a29788a (127.15.250.131:34037)
I20260808 07:03:29.927901 26419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97901242097c446ebf1d046fe86d0800" candidate_uuid: "117e71d831de49c89d100e8d430e6f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b4106f2b661f4faaa255d5636a29788a"
I20260808 07:03:29.927985 26419 raft_consensus.cc:3037] T 97901242097c446ebf1d046fe86d0800 P b4106f2b661f4faaa255d5636a29788a [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.928013 26151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97901242097c446ebf1d046fe86d0800" candidate_uuid: "117e71d831de49c89d100e8d430e6f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9db3325ac3e244919f866a2e04b9b9a0"
I20260808 07:03:29.928085 26151 raft_consensus.cc:3037] T 97901242097c446ebf1d046fe86d0800 P 9db3325ac3e244919f866a2e04b9b9a0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.928669 26419 raft_consensus.cc:2454] T 97901242097c446ebf1d046fe86d0800 P b4106f2b661f4faaa255d5636a29788a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 117e71d831de49c89d100e8d430e6f91 in term 1.
I20260808 07:03:29.928681 26151 raft_consensus.cc:2454] T 97901242097c446ebf1d046fe86d0800 P 9db3325ac3e244919f866a2e04b9b9a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 117e71d831de49c89d100e8d430e6f91 in term 1.
I20260808 07:03:29.928850 26220 leader_election.cc:302] T 97901242097c446ebf1d046fe86d0800 P 117e71d831de49c89d100e8d430e6f91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 117e71d831de49c89d100e8d430e6f91, 9db3325ac3e244919f866a2e04b9b9a0; no voters:
I20260808 07:03:29.928933 26508 raft_consensus.cc:2781] T 97901242097c446ebf1d046fe86d0800 P 117e71d831de49c89d100e8d430e6f91 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:29.928988 26508 raft_consensus.cc:686] T 97901242097c446ebf1d046fe86d0800 P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: Becoming Leader. State: Replica: 117e71d831de49c89d100e8d430e6f91, State: Running, Role: LEADER
I20260808 07:03:29.929033 26508 consensus_queue.cc:237] T 97901242097c446ebf1d046fe86d0800 P 117e71d831de49c89d100e8d430e6f91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.929589 26006 catalog_manager.cc:5495] T 97901242097c446ebf1d046fe86d0800 P 117e71d831de49c89d100e8d430e6f91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 117e71d831de49c89d100e8d430e6f91 (127.15.250.130). New cstate: current_term: 1 leader_uuid: "117e71d831de49c89d100e8d430e6f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } health_report { overall_health: UNKNOWN } } }
I20260808 07:03:29.931512 26286 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f64d7ac8e8894a81bbbe7d9f364f9ea6"
dest_uuid: "117e71d831de49c89d100e8d430e6f91"
from {username='slave'} at 127.0.0.1:43438
I20260808 07:03:29.931569 26286 raft_consensus.cc:482] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 07:03:29.931588 26286 raft_consensus.cc:3037] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 117e71d831de49c89d100e8d430e6f91 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.932034 26286 raft_consensus.cc:504] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 117e71d831de49c89d100e8d430e6f91 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.932144 26286 leader_election.cc:288] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 117e71d831de49c89d100e8d430e6f91 [CANDIDATE]: Term 1 election: Requested vote from peers 9db3325ac3e244919f866a2e04b9b9a0 (127.15.250.129:42123), b4106f2b661f4faaa255d5636a29788a (127.15.250.131:34037)
I20260808 07:03:29.932351 26151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f64d7ac8e8894a81bbbe7d9f364f9ea6" candidate_uuid: "117e71d831de49c89d100e8d430e6f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9db3325ac3e244919f866a2e04b9b9a0"
I20260808 07:03:29.932380 26419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f64d7ac8e8894a81bbbe7d9f364f9ea6" candidate_uuid: "117e71d831de49c89d100e8d430e6f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b4106f2b661f4faaa255d5636a29788a"
I20260808 07:03:29.932432 26151 raft_consensus.cc:3037] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 9db3325ac3e244919f866a2e04b9b9a0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.932478 26419 raft_consensus.cc:3037] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P b4106f2b661f4faaa255d5636a29788a [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:03:29.932987 26151 raft_consensus.cc:2454] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 9db3325ac3e244919f866a2e04b9b9a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 117e71d831de49c89d100e8d430e6f91 in term 1.
I20260808 07:03:29.933060 26419 raft_consensus.cc:2454] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P b4106f2b661f4faaa255d5636a29788a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 117e71d831de49c89d100e8d430e6f91 in term 1.
I20260808 07:03:29.933143 26220 leader_election.cc:302] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 117e71d831de49c89d100e8d430e6f91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 117e71d831de49c89d100e8d430e6f91, 9db3325ac3e244919f866a2e04b9b9a0; no voters:
I20260808 07:03:29.933233 26508 raft_consensus.cc:2781] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 117e71d831de49c89d100e8d430e6f91 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:03:29.933290 26508 raft_consensus.cc:686] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: Becoming Leader. State: Replica: 117e71d831de49c89d100e8d430e6f91, State: Running, Role: LEADER
I20260808 07:03:29.933329 26508 consensus_queue.cc:237] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 117e71d831de49c89d100e8d430e6f91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.933917 26006 catalog_manager.cc:5495] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 117e71d831de49c89d100e8d430e6f91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 117e71d831de49c89d100e8d430e6f91 (127.15.250.130). New cstate: current_term: 1 leader_uuid: "117e71d831de49c89d100e8d430e6f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } health_report { overall_health: UNKNOWN } } }
I20260808 07:03:29.935237 26259 tablet_service.cc:1514] Processing DeleteTablet for tablet c3f1db05fe244bae87a077fba49f764d with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:43438
I20260808 07:03:29.935636 26537 tablet_replica.cc:330] T c3f1db05fe244bae87a077fba49f764d P 117e71d831de49c89d100e8d430e6f91: stopping tablet replica
I20260808 07:03:29.935762 26537 raft_consensus.cc:2229] T c3f1db05fe244bae87a077fba49f764d P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: Raft consensus shutting down.
I20260808 07:03:29.935851 26537 pending_rounds.cc:62] T c3f1db05fe244bae87a077fba49f764d P 117e71d831de49c89d100e8d430e6f91: Trying to abort 1 pending ops.
I20260808 07:03:29.935876 26537 pending_rounds.cc:68] T c3f1db05fe244bae87a077fba49f764d P 117e71d831de49c89d100e8d430e6f91: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316163010123464704 op_type: NO_OP noop_request { }
I20260808 07:03:29.935925 26537 raft_consensus.cc:2866] T c3f1db05fe244bae87a077fba49f764d P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260808 07:03:29.935950 26537 raft_consensus.cc:2258] T c3f1db05fe244bae87a077fba49f764d P 117e71d831de49c89d100e8d430e6f91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 07:03:29.936393 26537 ts_tablet_manager.cc:1905] T c3f1db05fe244bae87a077fba49f764d P 117e71d831de49c89d100e8d430e6f91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 07:03:29.938016 26537 ts_tablet_manager.cc:1918] T c3f1db05fe244bae87a077fba49f764d P 117e71d831de49c89d100e8d430e6f91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260808 07:03:29.938079 26537 log.cc:1196] T c3f1db05fe244bae87a077fba49f764d P 117e71d831de49c89d100e8d430e6f91: Deleting WAL directory at /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/wals/c3f1db05fe244bae87a077fba49f764d
I20260808 07:03:29.939726 26259 tablet_service.cc:1514] Processing DeleteTablet for tablet 55dd567790424028b86b3883a524be4a with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:43438
I20260808 07:03:29.939827 26537 tablet_replica.cc:330] T 55dd567790424028b86b3883a524be4a P 117e71d831de49c89d100e8d430e6f91: stopping tablet replica
I20260808 07:03:29.939916 26537 raft_consensus.cc:2229] T 55dd567790424028b86b3883a524be4a P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: Raft consensus shutting down.
I20260808 07:03:29.939960 26537 pending_rounds.cc:62] T 55dd567790424028b86b3883a524be4a P 117e71d831de49c89d100e8d430e6f91: Trying to abort 1 pending ops.
I20260808 07:03:29.939980 26537 pending_rounds.cc:68] T 55dd567790424028b86b3883a524be4a P 117e71d831de49c89d100e8d430e6f91: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316163010154950656 op_type: NO_OP noop_request { }
I20260808 07:03:29.940016 26537 raft_consensus.cc:2866] T 55dd567790424028b86b3883a524be4a P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260808 07:03:29.940040 26537 raft_consensus.cc:2258] T 55dd567790424028b86b3883a524be4a P 117e71d831de49c89d100e8d430e6f91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 07:03:29.940354 26537 ts_tablet_manager.cc:1905] T 55dd567790424028b86b3883a524be4a P 117e71d831de49c89d100e8d430e6f91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 07:03:29.941473 26537 ts_tablet_manager.cc:1918] T 55dd567790424028b86b3883a524be4a P 117e71d831de49c89d100e8d430e6f91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260808 07:03:29.941565 26537 log.cc:1196] T 55dd567790424028b86b3883a524be4a P 117e71d831de49c89d100e8d430e6f91: Deleting WAL directory at /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/wals/55dd567790424028b86b3883a524be4a
I20260808 07:03:29.943156 26259 tablet_service.cc:1514] Processing DeleteTablet for tablet e0d8f0133ef04fbcb07c3e93ce4a8382 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:43438
I20260808 07:03:29.943292 26537 tablet_replica.cc:330] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 117e71d831de49c89d100e8d430e6f91: stopping tablet replica
I20260808 07:03:29.943349 26537 raft_consensus.cc:2229] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: Raft consensus shutting down.
I20260808 07:03:29.943389 26537 pending_rounds.cc:62] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 117e71d831de49c89d100e8d430e6f91: Trying to abort 1 pending ops.
I20260808 07:03:29.943406 26537 pending_rounds.cc:68] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 117e71d831de49c89d100e8d430e6f91: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316163010172645376 op_type: NO_OP noop_request { }
I20260808 07:03:29.943444 26537 raft_consensus.cc:2866] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260808 07:03:29.943466 26537 raft_consensus.cc:2258] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 117e71d831de49c89d100e8d430e6f91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 07:03:29.943683 26537 ts_tablet_manager.cc:1905] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 117e71d831de49c89d100e8d430e6f91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 07:03:29.944828 26537 ts_tablet_manager.cc:1918] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 117e71d831de49c89d100e8d430e6f91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260808 07:03:29.944880 26537 log.cc:1196] T e0d8f0133ef04fbcb07c3e93ce4a8382 P 117e71d831de49c89d100e8d430e6f91: Deleting WAL directory at /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/wals/e0d8f0133ef04fbcb07c3e93ce4a8382
I20260808 07:03:29.946262 26259 tablet_service.cc:1514] Processing DeleteTablet for tablet 051057f218174b81bf33ba8a5834e9da with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:43438
I20260808 07:03:29.946360 26537 tablet_replica.cc:330] T 051057f218174b81bf33ba8a5834e9da P 117e71d831de49c89d100e8d430e6f91: stopping tablet replica
I20260808 07:03:29.946410 26537 raft_consensus.cc:2229] T 051057f218174b81bf33ba8a5834e9da P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: Raft consensus shutting down.
I20260808 07:03:29.946446 26537 pending_rounds.cc:62] T 051057f218174b81bf33ba8a5834e9da P 117e71d831de49c89d100e8d430e6f91: Trying to abort 1 pending ops.
I20260808 07:03:29.946463 26537 pending_rounds.cc:68] T 051057f218174b81bf33ba8a5834e9da P 117e71d831de49c89d100e8d430e6f91: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316163010188410880 op_type: NO_OP noop_request { }
I20260808 07:03:29.946498 26537 raft_consensus.cc:2866] T 051057f218174b81bf33ba8a5834e9da P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260808 07:03:29.946513 26537 raft_consensus.cc:2258] T 051057f218174b81bf33ba8a5834e9da P 117e71d831de49c89d100e8d430e6f91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 07:03:29.946786 26537 ts_tablet_manager.cc:1905] T 051057f218174b81bf33ba8a5834e9da P 117e71d831de49c89d100e8d430e6f91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 07:03:29.948036 26537 ts_tablet_manager.cc:1918] T 051057f218174b81bf33ba8a5834e9da P 117e71d831de49c89d100e8d430e6f91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260808 07:03:29.948087 26537 log.cc:1196] T 051057f218174b81bf33ba8a5834e9da P 117e71d831de49c89d100e8d430e6f91: Deleting WAL directory at /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/wals/051057f218174b81bf33ba8a5834e9da
I20260808 07:03:29.949536 26259 tablet_service.cc:1514] Processing DeleteTablet for tablet dad0a14869924e4a895536588073b33b with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:43438
I20260808 07:03:29.949626 26537 tablet_replica.cc:330] T dad0a14869924e4a895536588073b33b P 117e71d831de49c89d100e8d430e6f91: stopping tablet replica
I20260808 07:03:29.949676 26537 raft_consensus.cc:2229] T dad0a14869924e4a895536588073b33b P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: Raft consensus shutting down.
I20260808 07:03:29.949714 26537 pending_rounds.cc:62] T dad0a14869924e4a895536588073b33b P 117e71d831de49c89d100e8d430e6f91: Trying to abort 1 pending ops.
I20260808 07:03:29.949731 26537 pending_rounds.cc:68] T dad0a14869924e4a895536588073b33b P 117e71d831de49c89d100e8d430e6f91: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316163010206535680 op_type: NO_OP noop_request { }
I20260808 07:03:29.949777 26537 raft_consensus.cc:2866] T dad0a14869924e4a895536588073b33b P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260808 07:03:29.949793 26537 raft_consensus.cc:2258] T dad0a14869924e4a895536588073b33b P 117e71d831de49c89d100e8d430e6f91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 07:03:29.950016 26537 ts_tablet_manager.cc:1905] T dad0a14869924e4a895536588073b33b P 117e71d831de49c89d100e8d430e6f91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 07:03:29.951660 26537 ts_tablet_manager.cc:1918] T dad0a14869924e4a895536588073b33b P 117e71d831de49c89d100e8d430e6f91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260808 07:03:29.951710 26537 log.cc:1196] T dad0a14869924e4a895536588073b33b P 117e71d831de49c89d100e8d430e6f91: Deleting WAL directory at /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/wals/dad0a14869924e4a895536588073b33b
I20260808 07:03:29.953109 26286 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1733039f2bdd4c4fa9a47522cbdefe9e"
dest_uuid: "117e71d831de49c89d100e8d430e6f91"
from {username='slave'} at 127.0.0.1:43438
I20260808 07:03:29.953223 26286 raft_consensus.cc:580] T 1733039f2bdd4c4fa9a47522cbdefe9e P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: Received request to step down
I20260808 07:03:29.953243 26286 raft_consensus.cc:3032] T 1733039f2bdd4c4fa9a47522cbdefe9e P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 07:03:29.953296 26286 raft_consensus.cc:729] T 1733039f2bdd4c4fa9a47522cbdefe9e P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 117e71d831de49c89d100e8d430e6f91, State: Running, Role: LEADER
I20260808 07:03:29.953331 26286 consensus_queue.cc:260] T 1733039f2bdd4c4fa9a47522cbdefe9e P 117e71d831de49c89d100e8d430e6f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.953401 26286 raft_consensus.cc:3037] T 1733039f2bdd4c4fa9a47522cbdefe9e P 117e71d831de49c89d100e8d430e6f91 [term 1 FOLLOWER]: Advancing to term 2
I20260808 07:03:29.954607 26259 tablet_service.cc:1514] Processing DeleteTablet for tablet 1733039f2bdd4c4fa9a47522cbdefe9e with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:43438
I20260808 07:03:29.954707 26537 tablet_replica.cc:330] T 1733039f2bdd4c4fa9a47522cbdefe9e P 117e71d831de49c89d100e8d430e6f91: stopping tablet replica
I20260808 07:03:29.954789 26537 raft_consensus.cc:2229] T 1733039f2bdd4c4fa9a47522cbdefe9e P 117e71d831de49c89d100e8d430e6f91 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 07:03:29.954838 26537 pending_rounds.cc:62] T 1733039f2bdd4c4fa9a47522cbdefe9e P 117e71d831de49c89d100e8d430e6f91: Trying to abort 1 pending ops.
I20260808 07:03:29.954880 26537 pending_rounds.cc:68] T 1733039f2bdd4c4fa9a47522cbdefe9e P 117e71d831de49c89d100e8d430e6f91: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316163010222661632 op_type: NO_OP noop_request { }
I20260808 07:03:29.954954 26537 raft_consensus.cc:2866] T 1733039f2bdd4c4fa9a47522cbdefe9e P 117e71d831de49c89d100e8d430e6f91 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260808 07:03:29.954996 26537 raft_consensus.cc:2258] T 1733039f2bdd4c4fa9a47522cbdefe9e P 117e71d831de49c89d100e8d430e6f91 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 07:03:29.955286 26537 ts_tablet_manager.cc:1905] T 1733039f2bdd4c4fa9a47522cbdefe9e P 117e71d831de49c89d100e8d430e6f91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 07:03:29.956679 26537 ts_tablet_manager.cc:1918] T 1733039f2bdd4c4fa9a47522cbdefe9e P 117e71d831de49c89d100e8d430e6f91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260808 07:03:29.956732 26537 log.cc:1196] T 1733039f2bdd4c4fa9a47522cbdefe9e P 117e71d831de49c89d100e8d430e6f91: Deleting WAL directory at /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/wals/1733039f2bdd4c4fa9a47522cbdefe9e
I20260808 07:03:29.958263 26286 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0080f8a2994e4252802ea6c5a6f4863b"
dest_uuid: "117e71d831de49c89d100e8d430e6f91"
from {username='slave'} at 127.0.0.1:43438
I20260808 07:03:29.958331 26286 raft_consensus.cc:580] T 0080f8a2994e4252802ea6c5a6f4863b P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: Received request to step down
I20260808 07:03:29.958348 26286 raft_consensus.cc:3032] T 0080f8a2994e4252802ea6c5a6f4863b P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 07:03:29.958360 26286 raft_consensus.cc:729] T 0080f8a2994e4252802ea6c5a6f4863b P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 117e71d831de49c89d100e8d430e6f91, State: Running, Role: LEADER
I20260808 07:03:29.958387 26286 consensus_queue.cc:260] T 0080f8a2994e4252802ea6c5a6f4863b P 117e71d831de49c89d100e8d430e6f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.958441 26286 raft_consensus.cc:3037] T 0080f8a2994e4252802ea6c5a6f4863b P 117e71d831de49c89d100e8d430e6f91 [term 1 FOLLOWER]: Advancing to term 2
I20260808 07:03:29.959630 26259 tablet_service.cc:1514] Processing DeleteTablet for tablet 0080f8a2994e4252802ea6c5a6f4863b with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:43438
I20260808 07:03:29.959717 26537 tablet_replica.cc:330] T 0080f8a2994e4252802ea6c5a6f4863b P 117e71d831de49c89d100e8d430e6f91: stopping tablet replica
I20260808 07:03:29.959765 26537 raft_consensus.cc:2229] T 0080f8a2994e4252802ea6c5a6f4863b P 117e71d831de49c89d100e8d430e6f91 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 07:03:29.959793 26537 pending_rounds.cc:62] T 0080f8a2994e4252802ea6c5a6f4863b P 117e71d831de49c89d100e8d430e6f91: Trying to abort 1 pending ops.
I20260808 07:03:29.959810 26537 pending_rounds.cc:68] T 0080f8a2994e4252802ea6c5a6f4863b P 117e71d831de49c89d100e8d430e6f91: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316163010238693376 op_type: NO_OP noop_request { }
I20260808 07:03:29.959846 26537 raft_consensus.cc:2866] T 0080f8a2994e4252802ea6c5a6f4863b P 117e71d831de49c89d100e8d430e6f91 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260808 07:03:29.959859 26537 raft_consensus.cc:2258] T 0080f8a2994e4252802ea6c5a6f4863b P 117e71d831de49c89d100e8d430e6f91 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 07:03:29.960088 26537 ts_tablet_manager.cc:1905] T 0080f8a2994e4252802ea6c5a6f4863b P 117e71d831de49c89d100e8d430e6f91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 07:03:29.961392 26537 ts_tablet_manager.cc:1918] T 0080f8a2994e4252802ea6c5a6f4863b P 117e71d831de49c89d100e8d430e6f91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260808 07:03:29.961443 26537 log.cc:1196] T 0080f8a2994e4252802ea6c5a6f4863b P 117e71d831de49c89d100e8d430e6f91: Deleting WAL directory at /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/wals/0080f8a2994e4252802ea6c5a6f4863b
I20260808 07:03:29.962788 26286 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3f58a7559c834237966455a46e2c6d02"
dest_uuid: "117e71d831de49c89d100e8d430e6f91"
from {username='slave'} at 127.0.0.1:43438
I20260808 07:03:29.962846 26286 raft_consensus.cc:580] T 3f58a7559c834237966455a46e2c6d02 P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: Received request to step down
I20260808 07:03:29.962863 26286 raft_consensus.cc:3032] T 3f58a7559c834237966455a46e2c6d02 P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 07:03:29.962874 26286 raft_consensus.cc:729] T 3f58a7559c834237966455a46e2c6d02 P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 117e71d831de49c89d100e8d430e6f91, State: Running, Role: LEADER
I20260808 07:03:29.962920 26286 consensus_queue.cc:260] T 3f58a7559c834237966455a46e2c6d02 P 117e71d831de49c89d100e8d430e6f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.962970 26286 raft_consensus.cc:3037] T 3f58a7559c834237966455a46e2c6d02 P 117e71d831de49c89d100e8d430e6f91 [term 1 FOLLOWER]: Advancing to term 2
I20260808 07:03:29.964097 26259 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f58a7559c834237966455a46e2c6d02 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:43438
I20260808 07:03:29.964183 26537 tablet_replica.cc:330] T 3f58a7559c834237966455a46e2c6d02 P 117e71d831de49c89d100e8d430e6f91: stopping tablet replica
I20260808 07:03:29.964232 26537 raft_consensus.cc:2229] T 3f58a7559c834237966455a46e2c6d02 P 117e71d831de49c89d100e8d430e6f91 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 07:03:29.964294 26537 pending_rounds.cc:62] T 3f58a7559c834237966455a46e2c6d02 P 117e71d831de49c89d100e8d430e6f91: Trying to abort 1 pending ops.
I20260808 07:03:29.964313 26537 pending_rounds.cc:68] T 3f58a7559c834237966455a46e2c6d02 P 117e71d831de49c89d100e8d430e6f91: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316163010254254080 op_type: NO_OP noop_request { }
I20260808 07:03:29.964357 26537 raft_consensus.cc:2866] T 3f58a7559c834237966455a46e2c6d02 P 117e71d831de49c89d100e8d430e6f91 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260808 07:03:29.964381 26537 raft_consensus.cc:2258] T 3f58a7559c834237966455a46e2c6d02 P 117e71d831de49c89d100e8d430e6f91 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 07:03:29.964622 26537 ts_tablet_manager.cc:1905] T 3f58a7559c834237966455a46e2c6d02 P 117e71d831de49c89d100e8d430e6f91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 07:03:29.965746 26537 ts_tablet_manager.cc:1918] T 3f58a7559c834237966455a46e2c6d02 P 117e71d831de49c89d100e8d430e6f91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260808 07:03:29.965797 26537 log.cc:1196] T 3f58a7559c834237966455a46e2c6d02 P 117e71d831de49c89d100e8d430e6f91: Deleting WAL directory at /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/wals/3f58a7559c834237966455a46e2c6d02
I20260808 07:03:29.967187 26286 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "97901242097c446ebf1d046fe86d0800"
dest_uuid: "117e71d831de49c89d100e8d430e6f91"
from {username='slave'} at 127.0.0.1:43438
I20260808 07:03:29.967242 26286 raft_consensus.cc:580] T 97901242097c446ebf1d046fe86d0800 P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: Received request to step down
I20260808 07:03:29.967260 26286 raft_consensus.cc:3032] T 97901242097c446ebf1d046fe86d0800 P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 07:03:29.967272 26286 raft_consensus.cc:729] T 97901242097c446ebf1d046fe86d0800 P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 117e71d831de49c89d100e8d430e6f91, State: Running, Role: LEADER
I20260808 07:03:29.967299 26286 consensus_queue.cc:260] T 97901242097c446ebf1d046fe86d0800 P 117e71d831de49c89d100e8d430e6f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.967351 26286 raft_consensus.cc:3037] T 97901242097c446ebf1d046fe86d0800 P 117e71d831de49c89d100e8d430e6f91 [term 1 FOLLOWER]: Advancing to term 2
I20260808 07:03:29.968564 26259 tablet_service.cc:1514] Processing DeleteTablet for tablet 97901242097c446ebf1d046fe86d0800 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:43438
I20260808 07:03:29.968650 26537 tablet_replica.cc:330] T 97901242097c446ebf1d046fe86d0800 P 117e71d831de49c89d100e8d430e6f91: stopping tablet replica
I20260808 07:03:29.968705 26537 raft_consensus.cc:2229] T 97901242097c446ebf1d046fe86d0800 P 117e71d831de49c89d100e8d430e6f91 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 07:03:29.968734 26537 pending_rounds.cc:62] T 97901242097c446ebf1d046fe86d0800 P 117e71d831de49c89d100e8d430e6f91: Trying to abort 1 pending ops.
I20260808 07:03:29.968746 26537 pending_rounds.cc:68] T 97901242097c446ebf1d046fe86d0800 P 117e71d831de49c89d100e8d430e6f91: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316163010269634560 op_type: NO_OP noop_request { }
I20260808 07:03:29.968782 26537 raft_consensus.cc:2866] T 97901242097c446ebf1d046fe86d0800 P 117e71d831de49c89d100e8d430e6f91 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260808 07:03:29.968797 26537 raft_consensus.cc:2258] T 97901242097c446ebf1d046fe86d0800 P 117e71d831de49c89d100e8d430e6f91 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 07:03:29.969024 26537 ts_tablet_manager.cc:1905] T 97901242097c446ebf1d046fe86d0800 P 117e71d831de49c89d100e8d430e6f91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 07:03:29.970149 26537 ts_tablet_manager.cc:1918] T 97901242097c446ebf1d046fe86d0800 P 117e71d831de49c89d100e8d430e6f91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260808 07:03:29.970211 26537 log.cc:1196] T 97901242097c446ebf1d046fe86d0800 P 117e71d831de49c89d100e8d430e6f91: Deleting WAL directory at /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/wals/97901242097c446ebf1d046fe86d0800
I20260808 07:03:29.971462 26286 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f64d7ac8e8894a81bbbe7d9f364f9ea6"
dest_uuid: "117e71d831de49c89d100e8d430e6f91"
from {username='slave'} at 127.0.0.1:43438
I20260808 07:03:29.971518 26286 raft_consensus.cc:580] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: Received request to step down
I20260808 07:03:29.971535 26286 raft_consensus.cc:3032] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 07:03:29.971547 26286 raft_consensus.cc:729] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 117e71d831de49c89d100e8d430e6f91 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 117e71d831de49c89d100e8d430e6f91, State: Running, Role: LEADER
I20260808 07:03:29.971573 26286 consensus_queue.cc:260] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 117e71d831de49c89d100e8d430e6f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db3325ac3e244919f866a2e04b9b9a0" member_type: VOTER last_known_addr { host: "127.15.250.129" port: 42123 } } peers { permanent_uuid: "117e71d831de49c89d100e8d430e6f91" member_type: VOTER last_known_addr { host: "127.15.250.130" port: 36973 } } peers { permanent_uuid: "b4106f2b661f4faaa255d5636a29788a" member_type: VOTER last_known_addr { host: "127.15.250.131" port: 34037 } }
I20260808 07:03:29.971621 26286 raft_consensus.cc:3037] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 117e71d831de49c89d100e8d430e6f91 [term 1 FOLLOWER]: Advancing to term 2
I20260808 07:03:29.972811 26259 tablet_service.cc:1514] Processing DeleteTablet for tablet f64d7ac8e8894a81bbbe7d9f364f9ea6 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:43438
I20260808 07:03:29.972900 26537 tablet_replica.cc:330] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 117e71d831de49c89d100e8d430e6f91: stopping tablet replica
I20260808 07:03:29.972959 26537 raft_consensus.cc:2229] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 117e71d831de49c89d100e8d430e6f91 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 07:03:29.972986 26537 pending_rounds.cc:62] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 117e71d831de49c89d100e8d430e6f91: Trying to abort 1 pending ops.
I20260808 07:03:29.972998 26537 pending_rounds.cc:68] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 117e71d831de49c89d100e8d430e6f91: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316163010287312896 op_type: NO_OP noop_request { }
I20260808 07:03:29.973030 26537 raft_consensus.cc:2866] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 117e71d831de49c89d100e8d430e6f91 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260808 07:03:29.973045 26537 raft_consensus.cc:2258] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 117e71d831de49c89d100e8d430e6f91 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 07:03:29.973327 26537 ts_tablet_manager.cc:1905] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 117e71d831de49c89d100e8d430e6f91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 07:03:29.974431 26537 ts_tablet_manager.cc:1918] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 117e71d831de49c89d100e8d430e6f91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260808 07:03:29.974478 26537 log.cc:1196] T f64d7ac8e8894a81bbbe7d9f364f9ea6 P 117e71d831de49c89d100e8d430e6f91: Deleting WAL directory at /tmp/dist-test-taskDkEqNX/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1786172446491335-16362-0/raft_consensus-itest-cluster/ts-1/wal/wals/f64d7ac8e8894a81bbbe7d9f364f9ea6
I20260808 07:03:29.975987 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 26067
I20260808 07:03:29.985147 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 26201
I20260808 07:03:29.990793 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 26335
I20260808 07:03:29.996170 16362 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDkEqNX/build/release/bin/kudu with pid 25975
2026-08-08T07:03:29Z chronyd exiting
[ OK ] NumReplicas/RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric/1 (797 ms)
[----------] 2 tests from NumReplicas/RaftConsensusNumLeadersMetricTest (1265 ms total)
[----------] Global test environment tear-down
[==========] 14 tests from 2 test suites ran. (163516 ms total)
[ PASSED ] 13 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] RaftConsensusElectionITest.TestNewLeaderCantResolvePeers
1 FAILED TEST
I20260808 07:03:30.020525 16362 logging.cc:424] LogThrottler /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 8 messages since previous log ~9 seconds ago